task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-hr79r-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-hr79r-kubeconfig [get-kubeconfig] Found admin password secret: cluster-hr79r-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-hr79r-password [get-kubeconfig] API Server URL: https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.35813af0-8a90-4d19-abf5-d2d9edb86f51.prod.konfluxeaas.com [clone-repo] rhoaieng-76590-enable-csr [clone-repo] https://github.com/mholder6/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 46% (1501/3241) Updating files: 47% (1524/3241) Updating files: 48% (1556/3241) Updating files: 49% (1589/3241) Updating files: 50% (1621/3241) Updating files: 51% (1653/3241) Updating files: 52% (1686/3241) Updating files: 53% (1718/3241) Updating files: 54% (1751/3241) Updating files: 55% (1783/3241) Updating files: 56% (1815/3241) Updating files: 57% (1848/3241) Updating files: 58% (1880/3241) Updating files: 59% (1913/3241) Updating files: 60% (1945/3241) Updating files: 61% (1978/3241) Updating files: 62% (2010/3241) Updating files: 63% (2042/3241) Updating files: 64% (2075/3241) Updating files: 65% (2107/3241) Updating files: 66% (2140/3241) Updating files: 67% (2172/3241) Updating files: 68% (2204/3241) Updating files: 69% (2237/3241) Updating files: 70% (2269/3241) Updating files: 71% (2302/3241) Updating files: 72% (2334/3241) Updating files: 73% (2366/3241) Updating files: 74% (2399/3241) Updating files: 75% (2431/3241) Updating files: 76% (2464/3241) Updating files: 77% (2496/3241) Updating files: 78% (2528/3241) Updating files: 79% (2561/3241) Updating files: 80% (2593/3241) Updating files: 81% (2626/3241) Updating files: 82% (2658/3241) Updating files: 83% (2691/3241) Updating files: 84% (2723/3241) Updating files: 85% (2755/3241) Updating files: 86% (2788/3241) Updating files: 87% (2820/3241) Updating files: 88% (2853/3241) Updating files: 89% (2885/3241) Updating files: 90% (2917/3241) Updating files: 91% (2950/3241) Updating files: 92% (2982/3241) Updating files: 93% (3015/3241) Updating files: 94% (3047/3241) Updating files: 95% (3079/3241) Updating files: 96% (3112/3241) Updating files: 97% (3144/3241) Updating files: 98% (3177/3241) Updating files: 99% (3209/3241) Updating files: 100% (3241/3241) Updating files: 100% (3241/3241), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:59116e393727edddfae1644c3d81197edfd10cc178aa036a7b2bffb50cf78240 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:59116e393727edddfae1644c3d81197edfd10cc178aa036a7b2bffb50cf78240 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:6b1a761f9a99cff87ede5b6d35aa00726047a46dad564bb0bdc84a05702b9ee9 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:6b1a761f9a99cff87ede5b6d35aa00726047a46dad564bb0bdc84a05702b9ee9 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:bb41cd61270b68652f31f0340f92e8fb4ad2a90b39ae98edc75bad022f0102b2 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:bb41cd61270b68652f31f0340f92e8fb4ad2a90b39ae98edc75bad022f0102b2 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:16acc6d59cc8f556c77d5ddde6c3a0999bf1b093cd9174dd79d24290082cc300 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:16acc6d59cc8f556c77d5ddde6c3a0999bf1b093cd9174dd79d24290082cc300 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:42f7b5da7a3bb4aecf39f72891f632714f3486c52caeb9aef675024ea5441c78 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:42f7b5da7a3bb4aecf39f72891f632714f3486c52caeb9aef675024ea5441c78 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1803 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1803 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1803 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.85s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 447ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 50ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-rcwgh 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-rcwgh condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-tnf9z 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-tnf9z condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-lqc9h 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-lqc9h condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:6b1a761f9a99cff87ede5b6d35aa00726047a46dad564bb0bdc84a05702b9ee9 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:42f7b5da7a3bb4aecf39f72891f632714f3486c52caeb9aef675024ea5441c78 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:59116e393727edddfae1644c3d81197edfd10cc178aa036a7b2bffb50cf78240 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:bb41cd61270b68652f31f0340f92e8fb4ad2a90b39ae98edc75bad022f0102b2 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:16acc6d59cc8f556c77d5ddde6c3a0999bf1b093cd9174dd79d24290082cc300 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:6b1a761f9a99cff87ede5b6d35aa00726047a46dad564bb0bdc84a05702b9ee9 [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:42f7b5da7a3bb4aecf39f72891f632714f3486c52caeb9aef675024ea5441c78 [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:59116e393727edddfae1644c3d81197edfd10cc178aa036a7b2bffb50cf78240 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:bb41cd61270b68652f31f0340f92e8fb4ad2a90b39ae98edc75bad022f0102b2 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:16acc6d59cc8f556c77d5ddde6c3a0999bf1b093cd9174dd79d24290082cc300 [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-68fcfb49c7-g5w7l 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-68fcfb49c7-g5w7l condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-dm88t 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-dm88t condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-dz27l 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-dz27l condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Pending). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-b9spq 1/1 Running 0 84s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-b9spq condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-b9spq 1/1 Running 0 93s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-b9spq condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-hr79r-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-hr79r-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-hr79r-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-hr79r-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-hr79r-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-hr79r-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Wed Jul 29 22:59:40 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-lgvjv condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-844cf74686-qq7rx" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-844cf74686-w286v condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:6b1a761f9a99cff87ede5b6d35aa00726047a46dad564bb0bdc84a05702b9ee9 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:6b1a761f9a99cff87ede5b6d35aa00726047a46dad564bb0bdc84a05702b9ee9 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:653968f8736ed750abb0f1615f7c7063753ac233552d3cfeb87360b42b0dc2e6 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:42f7b5da7a3bb4aecf39f72891f632714f3486c52caeb9aef675024ea5441c78 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:42f7b5da7a3bb4aecf39f72891f632714f3486c52caeb9aef675024ea5441c78 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.124.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'anyio': '4.9.0', 'xdist': '3.6.1', 'cov': '5.0.0', 'metadata': '3.1.1', 'json-report': '1.5.0', 'httpx': '0.30.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-0.23.8, anyio-4.9.0, xdist-3.6.1, cov-5.0.0, metadata-3.1.1, json-report-1.5.0, httpx-0.30.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] 2026-07-29 23:02:09.442 6643 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-29 23:02:09.443 6640 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-29 23:04:39.361 6640 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-29 23:04:39.361 6640 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-29 23:04:39.393 6640 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 23:04:39.393 6640 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 23:04:39.404 6640 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 23:04:39.404 6640 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-29 23:05:02.233 6640 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-29 23:05:02.233 6640 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-29 23:05:02.266 6640 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 23:05:02.266 6640 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 23:05:02.276 6640 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-29 23:05:02.276 6640 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw0] ERROR llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] ERROR llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw0] ERROR llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] ERROR llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [e2e-llm-inference-service] ==================================== ERRORS ==================================== [e2e-llm-inference-service] _ ERROR at setup of test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1721: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-managed-autoscale-keda-d-27e06c40' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-managed-autoscale-keda-d-27e06c40' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-managed-autoscale-keda-d-27e06c40', 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-managed-autoscale-keda-d-27e06c40', 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/llminferenceserviceconfigs/router-managed-autoscale-keda-d-27e06c40' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b'), ('plural', 'llminferenceserviceconfigs'), ('name', 'router-managed-autoscale-keda-d-27e06c40')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a.../e2e-test-llm-autoscaling-keda-deployment-b2150d0b/llminferenceserviceconfigs/router-managed-autoscale-keda-d-27e06c40' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a.../e2e-test-llm-autoscaling-keda-deployment-b2150d0b/llminferenceserviceconfigs/router-managed-autoscale-keda-d-27e06c40' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a.../e2e-test-llm-autoscaling-keda-deployment-b2150d0b/llminferenceserviceconfigs/router-managed-autoscale-keda-d-27e06c40' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (404) [e2e-llm-inference-service] E Reason: Not Found [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '19162eaf-5d34-4ade-8ac4-f89fe7fd7536', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '338'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"llminferenceserviceconfigs.serving.kserve.io \"router-managed-autoscale-keda-d-27e06c40\" not found","reason":"NotFound","details":{"name":"router-managed-autoscale-keda-d-27e06c40","group":"serving.kserve.io","kind":"llminferenceserviceconfigs"},"code":404} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(autouse=True) [e2e-llm-inference-service] def ensure_gateway_proxy_memory(request): [e2e-llm-inference-service] """After test setup creates gateways, patch them for proxy memory.""" [e2e-llm-inference-service] if not GATEWAY_PROXY_MEMORY: [e2e-llm-inference-service] return [e2e-llm-inference-service] [e2e-llm-inference-service] # Let test_case (llmisvc) create gateways first [e2e-llm-inference-service] [e2e-llm-inference-service] if "test_case" in request.fixturenames: [e2e-llm-inference-service] > request.getfixturevalue("test_case") [e2e-llm-inference-service] [e2e-llm-inference-service] common/gateway_proxy_istio.py:183: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] test_namespace = 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b' [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request, test_namespace): [e2e-llm-inference-service] tc = request.param [e2e-llm-inference-service] ns = test_namespace [e2e-llm-inference-service] [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] tc.namespace = ns [e2e-llm-inference-service] for peer in tc.peers: [e2e-llm-inference-service] peer.namespace = ns [e2e-llm-inference-service] [e2e-llm-inference-service] for func in tc.before_test: [e2e-llm-inference-service] func(tc) [e2e-llm-inference-service] [e2e-llm-inference-service] > _setup_test_case_service(kserve_client, tc, request.node.name, namespace=ns) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1611: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] tc = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro...g-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service=None, model_name='facebook/opt-125m') [e2e-llm-inference-service] test_node_name = 'test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda]' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b' [e2e-llm-inference-service] peer_index = None [e2e-llm-inference-service] [e2e-llm-inference-service] def _setup_test_case_service( [e2e-llm-inference-service] kserve_client, tc, test_node_name, namespace, peer_index=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Create LLMInferenceServiceConfigs and build the LLMInferenceService for a TestCase. [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a list of created config names for cleanup tracking. [e2e-llm-inference-service] """ [e2e-llm-inference-service] missing_refs = [ [e2e-llm-inference-service] ref for ref in tc.base_refs if ref not in LLMINFERENCESERVICE_CONFIGS [e2e-llm-inference-service] ] [e2e-llm-inference-service] if missing_refs: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Missing base_refs in LLMINFERENCESERVICE_CONFIGS: {missing_refs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] if not tc.service_name: [e2e-llm-inference-service] suffix = f"-peer-{peer_index}" if peer_index is not None else "" [e2e-llm-inference-service] tc.service_name = generate_service_name(test_node_name + suffix, tc.base_refs) [e2e-llm-inference-service] if tc.model_name == "default/model": [e2e-llm-inference-service] tc.model_name = _get_model_name_from_configs(tc.base_refs) [e2e-llm-inference-service] elif "{namespace}" in tc.model_name: [e2e-llm-inference-service] tc.model_name = tc.model_name.format(namespace=namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] created_configs = [] [e2e-llm-inference-service] unique_base_refs = [] [e2e-llm-inference-service] for base_ref in tc.base_refs: [e2e-llm-inference-service] unique_config_name = generate_k8s_safe_suffix(base_ref, [tc.service_name]) [e2e-llm-inference-service] unique_base_refs.append(unique_config_name) [e2e-llm-inference-service] [e2e-llm-inference-service] config = LLMINFERENCESERVICE_CONFIGS[base_ref] [e2e-llm-inference-service] spec = config(namespace) if callable(config) else copy.deepcopy(config) [e2e-llm-inference-service] [e2e-llm-inference-service] unique_config_body = { [e2e-llm-inference-service] "apiVersion": "serving.kserve.io/v1alpha1", [e2e-llm-inference-service] "kind": "LLMInferenceServiceConfig", [e2e-llm-inference-service] "metadata": { [e2e-llm-inference-service] "name": unique_config_name, [e2e-llm-inference-service] "namespace": namespace, [e2e-llm-inference-service] }, [e2e-llm-inference-service] "spec": spec, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] > _create_or_update_llmisvc_config(kserve_client, unique_config_body, namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1573: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llm_config["metadata"] = existing_config["metadata"] [e2e-llm-inference-service] [e2e-llm-inference-service] outputs = kserve_client.api_instance.replace_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] logger.info(f"✓ Successfully updated LLMInferenceServiceConfig {name}") [e2e-llm-inference-service] return outputs [e2e-llm-inference-service] [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] if e.status == 404: # Not found - create it [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"Resource not found, creating LLMInferenceServiceConfig {name}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] > outputs = kserve_client.api_instance.create_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1747: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:231: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...], 'au...t-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [...]}}}}}, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'body', [e2e-llm-inference-service] 'pretty', [e2e-llm-inference-service] 'dry_run', [e2e-llm-inference-service] 'field_manager', [e2e-llm-inference-service] 'field_validation' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method create_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'body' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('body' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['body'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `body` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-llm-inference-service] if 'dry_run' in local_var_params and local_var_params['dry_run'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('dryRun', local_var_params['dry_run'])) # noqa: E501 [e2e-llm-inference-service] if 'field_manager' in local_var_params and local_var_params['field_manager'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldManager', local_var_params['field_manager'])) # noqa: E501 [e2e-llm-inference-service] if 'field_validation' in local_var_params and local_var_params['field_validation'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldValidation', local_var_params['field_validation'])) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] if 'body' in local_var_params: [e2e-llm-inference-service] body_params = local_var_params['body'] [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:354: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/llminferenceserviceconfigs' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b'), ('plural', 'llminferenceserviceconfigs')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "HEAD": [e2e-llm-inference-service] return self.rest_client.HEAD(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "OPTIONS": [e2e-llm-inference-service] return self.rest_client.OPTIONS(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] elif method == "POST": [e2e-llm-inference-service] > return self.rest_client.POST(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:391: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/llminferenceserviceconfigs' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def POST(self, url, headers=None, query_params=None, post_params=None, [e2e-llm-inference-service] body=None, _preload_content=True, _request_timeout=None): [e2e-llm-inference-service] > return self.request("POST", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] body=body) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:279: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...yment-b2150d0b'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '77552e78-3fde-4050-aa6a-c4deedc15f86', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '701'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": EOF","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": EOF"}]},"code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] _ ERROR at setup of test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1721: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-managed-autoscale-hpa-lw-1aa98714' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-managed-autoscale-hpa-lw-1aa98714' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-managed-autoscale-hpa-lw-1aa98714', 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-managed-autoscale-hpa-lw-1aa98714', 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/llminferenceserviceconfigs/router-managed-autoscale-hpa-lw-1aa98714' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2'), ('plural', 'llminferenceserviceconfigs'), ('name', 'router-managed-autoscale-hpa-lw-1aa98714')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...mespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/llminferenceserviceconfigs/router-managed-autoscale-hpa-lw-1aa98714' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...mespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/llminferenceserviceconfigs/router-managed-autoscale-hpa-lw-1aa98714' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...mespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/llminferenceserviceconfigs/router-managed-autoscale-hpa-lw-1aa98714' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (404) [e2e-llm-inference-service] E Reason: Not Found [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a7a0c013-3c69-42aa-b3bf-51d822ff725f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '338'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"llminferenceserviceconfigs.serving.kserve.io \"router-managed-autoscale-hpa-lw-1aa98714\" not found","reason":"NotFound","details":{"name":"router-managed-autoscale-hpa-lw-1aa98714","group":"serving.kserve.io","kind":"llminferenceserviceconfigs"},"code":404} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(autouse=True) [e2e-llm-inference-service] def ensure_gateway_proxy_memory(request): [e2e-llm-inference-service] """After test setup creates gateways, patch them for proxy memory.""" [e2e-llm-inference-service] if not GATEWAY_PROXY_MEMORY: [e2e-llm-inference-service] return [e2e-llm-inference-service] [e2e-llm-inference-service] # Let test_case (llmisvc) create gateways first [e2e-llm-inference-service] [e2e-llm-inference-service] if "test_case" in request.fixturenames: [e2e-llm-inference-service] > request.getfixturevalue("test_case") [e2e-llm-inference-service] [e2e-llm-inference-service] common/gateway_proxy_istio.py:183: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] test_namespace = 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2' [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request, test_namespace): [e2e-llm-inference-service] tc = request.param [e2e-llm-inference-service] ns = test_namespace [e2e-llm-inference-service] [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] tc.namespace = ns [e2e-llm-inference-service] for peer in tc.peers: [e2e-llm-inference-service] peer.namespace = ns [e2e-llm-inference-service] [e2e-llm-inference-service] for func in tc.before_test: [e2e-llm-inference-service] func(tc) [e2e-llm-inference-service] [e2e-llm-inference-service] > _setup_test_case_service(kserve_client, tc, request.node.name, namespace=ns) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1611: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] tc = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer...toscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service=None, model_name='facebook/opt-125m') [e2e-llm-inference-service] test_node_name = 'test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa]' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', peer_index = None [e2e-llm-inference-service] [e2e-llm-inference-service] def _setup_test_case_service( [e2e-llm-inference-service] kserve_client, tc, test_node_name, namespace, peer_index=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Create LLMInferenceServiceConfigs and build the LLMInferenceService for a TestCase. [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a list of created config names for cleanup tracking. [e2e-llm-inference-service] """ [e2e-llm-inference-service] missing_refs = [ [e2e-llm-inference-service] ref for ref in tc.base_refs if ref not in LLMINFERENCESERVICE_CONFIGS [e2e-llm-inference-service] ] [e2e-llm-inference-service] if missing_refs: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Missing base_refs in LLMINFERENCESERVICE_CONFIGS: {missing_refs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] if not tc.service_name: [e2e-llm-inference-service] suffix = f"-peer-{peer_index}" if peer_index is not None else "" [e2e-llm-inference-service] tc.service_name = generate_service_name(test_node_name + suffix, tc.base_refs) [e2e-llm-inference-service] if tc.model_name == "default/model": [e2e-llm-inference-service] tc.model_name = _get_model_name_from_configs(tc.base_refs) [e2e-llm-inference-service] elif "{namespace}" in tc.model_name: [e2e-llm-inference-service] tc.model_name = tc.model_name.format(namespace=namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] created_configs = [] [e2e-llm-inference-service] unique_base_refs = [] [e2e-llm-inference-service] for base_ref in tc.base_refs: [e2e-llm-inference-service] unique_config_name = generate_k8s_safe_suffix(base_ref, [tc.service_name]) [e2e-llm-inference-service] unique_base_refs.append(unique_config_name) [e2e-llm-inference-service] [e2e-llm-inference-service] config = LLMINFERENCESERVICE_CONFIGS[base_ref] [e2e-llm-inference-service] spec = config(namespace) if callable(config) else copy.deepcopy(config) [e2e-llm-inference-service] [e2e-llm-inference-service] unique_config_body = { [e2e-llm-inference-service] "apiVersion": "serving.kserve.io/v1alpha1", [e2e-llm-inference-service] "kind": "LLMInferenceServiceConfig", [e2e-llm-inference-service] "metadata": { [e2e-llm-inference-service] "name": unique_config_name, [e2e-llm-inference-service] "namespace": namespace, [e2e-llm-inference-service] }, [e2e-llm-inference-service] "spec": spec, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] > _create_or_update_llmisvc_config(kserve_client, unique_config_body, namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1573: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llm_config["metadata"] = existing_config["metadata"] [e2e-llm-inference-service] [e2e-llm-inference-service] outputs = kserve_client.api_instance.replace_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] logger.info(f"✓ Successfully updated LLMInferenceServiceConfig {name}") [e2e-llm-inference-service] return outputs [e2e-llm-inference-service] [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] if e.status == 404: # Not found - create it [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"Resource not found, creating LLMInferenceServiceConfig {name}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] > outputs = kserve_client.api_instance.create_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1747: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:231: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...], 'au...s-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [...]}}}}}, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'body', [e2e-llm-inference-service] 'pretty', [e2e-llm-inference-service] 'dry_run', [e2e-llm-inference-service] 'field_manager', [e2e-llm-inference-service] 'field_validation' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method create_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'body' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('body' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['body'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `body` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-llm-inference-service] if 'dry_run' in local_var_params and local_var_params['dry_run'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('dryRun', local_var_params['dry_run'])) # noqa: E501 [e2e-llm-inference-service] if 'field_manager' in local_var_params and local_var_params['field_manager'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldManager', local_var_params['field_manager'])) # noqa: E501 [e2e-llm-inference-service] if 'field_validation' in local_var_params and local_var_params['field_validation'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldValidation', local_var_params['field_validation'])) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] if 'body' in local_var_params: [e2e-llm-inference-service] body_params = local_var_params['body'] [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:354: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/llminferenceserviceconfigs' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2'), ('plural', 'llminferenceserviceconfigs')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "HEAD": [e2e-llm-inference-service] return self.rest_client.HEAD(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "OPTIONS": [e2e-llm-inference-service] return self.rest_client.OPTIONS(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] elif method == "POST": [e2e-llm-inference-service] > return self.rest_client.POST(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:391: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/llminferenceserviceconfigs' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def POST(self, url, headers=None, query_params=None, post_params=None, [e2e-llm-inference-service] body=None, _preload_content=True, _request_timeout=None): [e2e-llm-inference-service] > return self.request("POST", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] body=body) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:279: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-d4cbcfd2'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '0aed1d32-84a8-4146-868a-907fbfa938df', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '701'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": EOF","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": EOF"}]},"code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] _ ERROR at setup of test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-lws-e541a132' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1721: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-lws-e541a132' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-managed-autoscale-keda-l-78828c4a' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-lws-e541a132' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-managed-autoscale-keda-l-78828c4a' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-managed-autoscale-keda-l-78828c4a', 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-managed-autoscale-keda-l-78828c4a', 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/llminferenceserviceconfigs/router-managed-autoscale-keda-l-78828c4a' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-keda-lws-e541a132'), ('plural', 'llminferenceserviceconfigs'), ('name', 'router-managed-autoscale-keda-l-78828c4a')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...espaces/e2e-test-llm-autoscaling-keda-lws-e541a132/llminferenceserviceconfigs/router-managed-autoscale-keda-l-78828c4a' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...espaces/e2e-test-llm-autoscaling-keda-lws-e541a132/llminferenceserviceconfigs/router-managed-autoscale-keda-l-78828c4a' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...espaces/e2e-test-llm-autoscaling-keda-lws-e541a132/llminferenceserviceconfigs/router-managed-autoscale-keda-l-78828c4a' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (404) [e2e-llm-inference-service] E Reason: Not Found [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '447f914d-c18f-488b-a9fb-93ffcd5927c7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:47 GMT', 'Content-Length': '338'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"llminferenceserviceconfigs.serving.kserve.io \"router-managed-autoscale-keda-l-78828c4a\" not found","reason":"NotFound","details":{"name":"router-managed-autoscale-keda-l-78828c4a","group":"serving.kserve.io","kind":"llminferenceserviceconfigs"},"code":404} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(autouse=True) [e2e-llm-inference-service] def ensure_gateway_proxy_memory(request): [e2e-llm-inference-service] """After test setup creates gateways, patch them for proxy memory.""" [e2e-llm-inference-service] if not GATEWAY_PROXY_MEMORY: [e2e-llm-inference-service] return [e2e-llm-inference-service] [e2e-llm-inference-service] # Let test_case (llmisvc) create gateways first [e2e-llm-inference-service] [e2e-llm-inference-service] if "test_case" in request.fixturenames: [e2e-llm-inference-service] > request.getfixturevalue("test_case") [e2e-llm-inference-service] [e2e-llm-inference-service] common/gateway_proxy_istio.py:183: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] test_namespace = 'e2e-test-llm-autoscaling-keda-lws-e541a132' [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request, test_namespace): [e2e-llm-inference-service] tc = request.param [e2e-llm-inference-service] ns = test_namespace [e2e-llm-inference-service] [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] tc.namespace = ns [e2e-llm-inference-service] for peer in tc.peers: [e2e-llm-inference-service] peer.namespace = ns [e2e-llm-inference-service] [e2e-llm-inference-service] for func in tc.before_test: [e2e-llm-inference-service] func(tc) [e2e-llm-inference-service] [e2e-llm-inference-service] > _setup_test_case_service(kserve_client, tc, request.node.name, namespace=ns) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1611: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] tc = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe...oscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service=None, model_name='facebook/opt-125m') [e2e-llm-inference-service] test_node_name = 'test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda]' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-lws-e541a132', peer_index = None [e2e-llm-inference-service] [e2e-llm-inference-service] def _setup_test_case_service( [e2e-llm-inference-service] kserve_client, tc, test_node_name, namespace, peer_index=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Create LLMInferenceServiceConfigs and build the LLMInferenceService for a TestCase. [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a list of created config names for cleanup tracking. [e2e-llm-inference-service] """ [e2e-llm-inference-service] missing_refs = [ [e2e-llm-inference-service] ref for ref in tc.base_refs if ref not in LLMINFERENCESERVICE_CONFIGS [e2e-llm-inference-service] ] [e2e-llm-inference-service] if missing_refs: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Missing base_refs in LLMINFERENCESERVICE_CONFIGS: {missing_refs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] if not tc.service_name: [e2e-llm-inference-service] suffix = f"-peer-{peer_index}" if peer_index is not None else "" [e2e-llm-inference-service] tc.service_name = generate_service_name(test_node_name + suffix, tc.base_refs) [e2e-llm-inference-service] if tc.model_name == "default/model": [e2e-llm-inference-service] tc.model_name = _get_model_name_from_configs(tc.base_refs) [e2e-llm-inference-service] elif "{namespace}" in tc.model_name: [e2e-llm-inference-service] tc.model_name = tc.model_name.format(namespace=namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] created_configs = [] [e2e-llm-inference-service] unique_base_refs = [] [e2e-llm-inference-service] for base_ref in tc.base_refs: [e2e-llm-inference-service] unique_config_name = generate_k8s_safe_suffix(base_ref, [tc.service_name]) [e2e-llm-inference-service] unique_base_refs.append(unique_config_name) [e2e-llm-inference-service] [e2e-llm-inference-service] config = LLMINFERENCESERVICE_CONFIGS[base_ref] [e2e-llm-inference-service] spec = config(namespace) if callable(config) else copy.deepcopy(config) [e2e-llm-inference-service] [e2e-llm-inference-service] unique_config_body = { [e2e-llm-inference-service] "apiVersion": "serving.kserve.io/v1alpha1", [e2e-llm-inference-service] "kind": "LLMInferenceServiceConfig", [e2e-llm-inference-service] "metadata": { [e2e-llm-inference-service] "name": unique_config_name, [e2e-llm-inference-service] "namespace": namespace, [e2e-llm-inference-service] }, [e2e-llm-inference-service] "spec": spec, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] > _create_or_update_llmisvc_config(kserve_client, unique_config_body, namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1573: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-lws-e541a132' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llm_config["metadata"] = existing_config["metadata"] [e2e-llm-inference-service] [e2e-llm-inference-service] outputs = kserve_client.api_instance.replace_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] logger.info(f"✓ Successfully updated LLMInferenceServiceConfig {name}") [e2e-llm-inference-service] return outputs [e2e-llm-inference-service] [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] if e.status == 404: # Not found - create it [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"Resource not found, creating LLMInferenceServiceConfig {name}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] > outputs = kserve_client.api_instance.create_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1747: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-lws-e541a132' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:231: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-keda-lws-e541a132' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...], 'au...s-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [...]}}}}}, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'body', [e2e-llm-inference-service] 'pretty', [e2e-llm-inference-service] 'dry_run', [e2e-llm-inference-service] 'field_manager', [e2e-llm-inference-service] 'field_validation' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method create_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'body' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('body' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['body'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `body` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-llm-inference-service] if 'dry_run' in local_var_params and local_var_params['dry_run'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('dryRun', local_var_params['dry_run'])) # noqa: E501 [e2e-llm-inference-service] if 'field_manager' in local_var_params and local_var_params['field_manager'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldManager', local_var_params['field_manager'])) # noqa: E501 [e2e-llm-inference-service] if 'field_validation' in local_var_params and local_var_params['field_validation'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldValidation', local_var_params['field_validation'])) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] if 'body' in local_var_params: [e2e-llm-inference-service] body_params = local_var_params['body'] [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:354: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/llminferenceserviceconfigs' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-keda-lws-e541a132'), ('plural', 'llminferenceserviceconfigs')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "HEAD": [e2e-llm-inference-service] return self.rest_client.HEAD(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "OPTIONS": [e2e-llm-inference-service] return self.rest_client.OPTIONS(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] elif method == "POST": [e2e-llm-inference-service] > return self.rest_client.POST(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:391: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/llminferenceserviceconfigs' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def POST(self, url, headers=None, query_params=None, post_params=None, [e2e-llm-inference-service] body=None, _preload_content=True, _request_timeout=None): [e2e-llm-inference-service] > return self.request("POST", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] body=body) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:279: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...a-lws-e541a132'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c623e33b-3903-4e23-ac45-ccd0ca8fa62e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:47 GMT', 'Content-Length': '701'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": EOF","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": EOF"}]},"code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] _ ERROR at setup of test_llm_autoscaling_cleanup_hpa[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1721: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-managed-autoscale-cleanu-e5a6b97f' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-managed-autoscale-cleanu-e5a6b97f' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-managed-autoscale-cleanu-e5a6b97f', 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-managed-autoscale-cleanu-e5a6b97f', 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/llminferenceserviceconfigs/router-managed-autoscale-cleanu-e5a6b97f' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40'), ('plural', 'llminferenceserviceconfigs'), ('name', 'router-managed-autoscale-cleanu-e5a6b97f')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...aces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/llminferenceserviceconfigs/router-managed-autoscale-cleanu-e5a6b97f' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...aces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/llminferenceserviceconfigs/router-managed-autoscale-cleanu-e5a6b97f' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...aces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/llminferenceserviceconfigs/router-managed-autoscale-cleanu-e5a6b97f' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (404) [e2e-llm-inference-service] E Reason: Not Found [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd9e2d8ad-f0dd-48ba-9b0c-b2e0bad93bf5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:47 GMT', 'Content-Length': '338'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"llminferenceserviceconfigs.serving.kserve.io \"router-managed-autoscale-cleanu-e5a6b97f\" not found","reason":"NotFound","details":{"name":"router-managed-autoscale-cleanu-e5a6b97f","group":"serving.kserve.io","kind":"llminferenceserviceconfigs"},"code":404} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(autouse=True) [e2e-llm-inference-service] def ensure_gateway_proxy_memory(request): [e2e-llm-inference-service] """After test setup creates gateways, patch them for proxy memory.""" [e2e-llm-inference-service] if not GATEWAY_PROXY_MEMORY: [e2e-llm-inference-service] return [e2e-llm-inference-service] [e2e-llm-inference-service] # Let test_case (llmisvc) create gateways first [e2e-llm-inference-service] [e2e-llm-inference-service] if "test_case" in request.fixturenames: [e2e-llm-inference-service] > request.getfixturevalue("test_case") [e2e-llm-inference-service] [e2e-llm-inference-service] common/gateway_proxy_istio.py:183: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] test_namespace = 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40' [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request, test_namespace): [e2e-llm-inference-service] tc = request.param [e2e-llm-inference-service] ns = test_namespace [e2e-llm-inference-service] [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] tc.namespace = ns [e2e-llm-inference-service] for peer in tc.peers: [e2e-llm-inference-service] peer.namespace = ns [e2e-llm-inference-service] [e2e-llm-inference-service] for func in tc.before_test: [e2e-llm-inference-service] func(tc) [e2e-llm-inference-service] [e2e-llm-inference-service] > _setup_test_case_service(kserve_client, tc, request.node.name, namespace=ns) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1611: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] tc = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom...aling-cleanup-hpa-a41f0e40', before_test=[], after_test=[], peers=[], llm_service=None, model_name='facebook/opt-125m') [e2e-llm-inference-service] test_node_name = 'test_llm_autoscaling_cleanup_hpa[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa]' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', peer_index = None [e2e-llm-inference-service] [e2e-llm-inference-service] def _setup_test_case_service( [e2e-llm-inference-service] kserve_client, tc, test_node_name, namespace, peer_index=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Create LLMInferenceServiceConfigs and build the LLMInferenceService for a TestCase. [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a list of created config names for cleanup tracking. [e2e-llm-inference-service] """ [e2e-llm-inference-service] missing_refs = [ [e2e-llm-inference-service] ref for ref in tc.base_refs if ref not in LLMINFERENCESERVICE_CONFIGS [e2e-llm-inference-service] ] [e2e-llm-inference-service] if missing_refs: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Missing base_refs in LLMINFERENCESERVICE_CONFIGS: {missing_refs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] if not tc.service_name: [e2e-llm-inference-service] suffix = f"-peer-{peer_index}" if peer_index is not None else "" [e2e-llm-inference-service] tc.service_name = generate_service_name(test_node_name + suffix, tc.base_refs) [e2e-llm-inference-service] if tc.model_name == "default/model": [e2e-llm-inference-service] tc.model_name = _get_model_name_from_configs(tc.base_refs) [e2e-llm-inference-service] elif "{namespace}" in tc.model_name: [e2e-llm-inference-service] tc.model_name = tc.model_name.format(namespace=namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] created_configs = [] [e2e-llm-inference-service] unique_base_refs = [] [e2e-llm-inference-service] for base_ref in tc.base_refs: [e2e-llm-inference-service] unique_config_name = generate_k8s_safe_suffix(base_ref, [tc.service_name]) [e2e-llm-inference-service] unique_base_refs.append(unique_config_name) [e2e-llm-inference-service] [e2e-llm-inference-service] config = LLMINFERENCESERVICE_CONFIGS[base_ref] [e2e-llm-inference-service] spec = config(namespace) if callable(config) else copy.deepcopy(config) [e2e-llm-inference-service] [e2e-llm-inference-service] unique_config_body = { [e2e-llm-inference-service] "apiVersion": "serving.kserve.io/v1alpha1", [e2e-llm-inference-service] "kind": "LLMInferenceServiceConfig", [e2e-llm-inference-service] "metadata": { [e2e-llm-inference-service] "name": unique_config_name, [e2e-llm-inference-service] "namespace": namespace, [e2e-llm-inference-service] }, [e2e-llm-inference-service] "spec": spec, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] > _create_or_update_llmisvc_config(kserve_client, unique_config_body, namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1573: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llm_config["metadata"] = existing_config["metadata"] [e2e-llm-inference-service] [e2e-llm-inference-service] outputs = kserve_client.api_instance.replace_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] logger.info(f"✓ Successfully updated LLMInferenceServiceConfig {name}") [e2e-llm-inference-service] return outputs [e2e-llm-inference-service] [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] if e.status == 404: # Not found - create it [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"Resource not found, creating LLMInferenceServiceConfig {name}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] > outputs = kserve_client.api_instance.create_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1747: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:231: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...], 'au...a-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [...]}}}}}, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'body', [e2e-llm-inference-service] 'pretty', [e2e-llm-inference-service] 'dry_run', [e2e-llm-inference-service] 'field_manager', [e2e-llm-inference-service] 'field_validation' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method create_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'body' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('body' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['body'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `body` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-llm-inference-service] if 'dry_run' in local_var_params and local_var_params['dry_run'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('dryRun', local_var_params['dry_run'])) # noqa: E501 [e2e-llm-inference-service] if 'field_manager' in local_var_params and local_var_params['field_manager'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldManager', local_var_params['field_manager'])) # noqa: E501 [e2e-llm-inference-service] if 'field_validation' in local_var_params and local_var_params['field_validation'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldValidation', local_var_params['field_validation'])) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] if 'body' in local_var_params: [e2e-llm-inference-service] body_params = local_var_params['body'] [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:354: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/llminferenceserviceconfigs' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40'), ('plural', 'llminferenceserviceconfigs')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "HEAD": [e2e-llm-inference-service] return self.rest_client.HEAD(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "OPTIONS": [e2e-llm-inference-service] return self.rest_client.OPTIONS(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] elif method == "POST": [e2e-llm-inference-service] > return self.rest_client.POST(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:391: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/llminferenceserviceconfigs' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def POST(self, url, headers=None, query_params=None, post_params=None, [e2e-llm-inference-service] body=None, _preload_content=True, _request_timeout=None): [e2e-llm-inference-service] > return self.request("POST", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] body=body) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:279: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-managed...p-hpa-a41f0e40'}, 'spec': {'router': {'gateway': {}, 'route': {}, 'scheduler': {'template': {'containers': [{...}]}}}}} [e2e-llm-inference-service] post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '4a3b755c-82da-4ec1-a025-b2415835951c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:47 GMT', 'Content-Length': '701'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": EOF","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": EOF"}]},"code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] name = 'autoscale-hpa-deploy' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4' [e2e-llm-inference-service] version = 'v1alpha1' [e2e-llm-inference-service] [e2e-llm-inference-service] def get_llmisvc( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] version=constants.KSERVE_V1ALPHA1_VERSION, [e2e-llm-inference-service] ): [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICE, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4' [e2e-llm-inference-service] plural = 'llminferenceservices', name = 'autoscale-hpa-deploy' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4' [e2e-llm-inference-service] plural = 'llminferenceservices', name = 'autoscale-hpa-deploy' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'autoscale-hpa-deploy', 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', 'plural': 'llminferenceservices', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'autoscale-hpa-deploy', 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', 'plural': 'llminferenceservices', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/llminferenceservices/autoscale-hpa-deploy' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4'), ('plural', 'llminferenceservices'), ('name', 'autoscale-hpa-deploy')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/llminferenceservices/autoscale-hpa-deploy' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/llminferenceservices/autoscale-hpa-deploy' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://ada79eace9eb4497397b552019af76bd-dd4019bb9b7ba4c9.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/llminferenceservices/autoscale-hpa-deploy' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '729fc6c4-b8f1-45c2-bbbb-3b9691a4f92f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T23:14:20.158810', start_time = 1785366860.1591873 [e2e-llm-inference-service] duration = 385.99627923965454, timestamp_end = '2026-07-29T23:20:46.155483' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8425854540> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] > out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1339: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] name = 'autoscale-hpa-deploy' [e2e-llm-inference-service] namespace = 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4' [e2e-llm-inference-service] version = 'v1alpha1' [e2e-llm-inference-service] [e2e-llm-inference-service] def get_llmisvc( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] version=constants.KSERVE_V1ALPHA1_VERSION, [e2e-llm-inference-service] ): [e2e-llm-inference-service] try: [e2e-llm-inference-service] return kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICE, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] > raise RuntimeError( [e2e-llm-inference-service] f"❌ Exception when calling CustomObjectsApi->" [e2e-llm-inference-service] f"get_namespaced_custom_object for LLMInferenceService: {e}" [e2e-llm-inference-service] ) from e [e2e-llm-inference-service] E RuntimeError: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '729fc6c4-b8f1-45c2-bbbb-3b9691a4f92f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1212: RuntimeError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-29T23:14:20.072560] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T23:14:20.085519] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T23:14:20.158648] end - ✅ in 0.073s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T23:14:20.158810] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:14:31Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:15:11Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:15:11Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:15:11Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:14:31Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:15:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:15:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:15:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-29T23:15:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:15:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 3: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-29T23:02:32Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T23:20:46.155483] end - ❌ 385.996s: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '729fc6c4-b8f1-45c2-bbbb-3b9691a4f92f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-29T23:20:46.155673] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [delete_llmisvc] [2026-07-29T23:20:46.165525] end - ❌ 0.009s: ❌ Exception when calling CustomObjectsApi->delete_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '8241a8f0-c45b-4fa5-9252-04ba2e98363c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] WARNING e2e.llmisvc.test_llm_autoscaling_wva:test_llm_autoscaling_wva.py:493 Failed to cleanup service: ❌ Exception when calling CustomObjectsApi->delete_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '8241a8f0-c45b-4fa5-9252-04ba2e98363c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-29T23:20:46.165694] end - ❌ 386.093s: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '729fc6c4-b8f1-45c2-bbbb-3b9691a4f92f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'f3c8a6d6-091a-4fdb-8e7a-4eccfe70cdac', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52445f87-6ff6-4459-b79d-336f1c75ac4c', 'Date': 'Wed, 29 Jul 2026 23:20:46 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-29T23:33:44.243857', start_time = 1785368024.2443533 [e2e-llm-inference-service] duration = 900.3548443317413, timestamp_end = '2026-07-29T23:48:44.599201' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fa840481800> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:36:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-48639af5 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-29T23:33:44.085530] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-48639af5', before_test=[ at 0x7fa8415974c0>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-29T23:33:44.097819] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-29T23:33:44.243640] end - ✅ in 0.146s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-29T23:33:44.243857] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'message': 'Inference Pool e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:36:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:36:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-29T23:48:44.599201] end - ❌ 900.355s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:36:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:36:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'e2e-test-llm-inference-service-48639af5' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:44Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:33:44Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:33:44Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T23:36:04Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] resourceVersion: '58850' [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a8fd6278e6ef447259ffca9489c1349b-1794000788.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a8fd6278e6ef447259ffca9489c1349b-1794000788.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a8fd6278e6ef447259ffca9489c1349b-1794000788.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:15Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:15Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:36:04Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:15Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:01Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:15Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:36Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:36Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:15Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a8fd6278e6ef447259ffca9489c1349b-1794000788.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l to ip-10-0-131-193.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:29 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-c2043d33-7945-4454-be43-31b1a1e806e0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:32 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.133.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:32 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:16acc6d59cc8f556c77d5ddde6c3a0999bf1b093cd9174dd79d24290082cc300" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:32 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:32 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:23 e2e-test-llm-inference-service-48639af5 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-dfd2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:42 e2e-test-llm-inference-service-48639af5 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:23 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:23 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:23 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-7cf5f9565-4vwfv_9208331e-2a6e-49f5-9460-cb2defff6921 Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-48639af5/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:26 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-7cf5f9565-4vwfv_9208331e-2a6e-49f5-9460-cb2defff6921 Normal ProvisioningSucceeded Successfully provisioned volume pvc-c2043d33-7945-4454-be43-31b1a1e806e0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl to ip-10-0-131-193.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:01 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-c2043d33-7945-4454-be43-31b1a1e806e0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:03 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.133.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:03 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:03 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:03 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:03 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:03 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:03 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:35:53 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-193.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.63:8001/health": dial tcp 10.133.0.63:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp to ip-10-0-128-242.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:01 e2e-test-llm-inference-service-48639af5 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-c2043d33-7945-4454-be43-31b1a1e806e0" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:54 e2e-test-llm-inference-service-48639af5 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-132820f03254470cd8a0146008974a5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:26 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:44:06 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:14 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:33:59 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv to ip-10-0-135-98.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:00 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:00 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-98.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:00 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-98.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-29 23:34:00 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-98.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 Flag --decoder-use-tls has been deprecated, use --enable-tls=decoder instead [e2e-llm-inference-service] Flag --prefiller-use-tls has been deprecated, use --enable-tls=prefiller instead [e2e-llm-inference-service] {"level":"info","ts":1785368043.380209,"msg":"Proxy starting","Built on":"","From Git SHA":"unknown"} [e2e-llm-inference-service] {"level":"info","ts":1785368043.3802345,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"e2e-test-llm-inference-service-48639af5\",\"InferencePoolName\":\"llmisvc-model-pvc-router-manage-e8706282-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1785368043.3815858,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"e2e-test-llm-inference-service-48639af5","poolName":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1785368043.395589,"logger":"allowlist-validator","msg":"InferencePool added","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368043.4042718,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785368043.4043481,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785368043.4821215,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785368043.4835606,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1785368043.4835818,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1785368043.489983,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368044.508331,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368053.1813953,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368073.3960874,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368073.4062274,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368073.4062788,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368103.396215,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368103.406036,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368103.4060977,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368133.3967786,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368133.4083798,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368133.4084506,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368163.1835139,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368163.2012684,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368163.396949,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368163.4046452,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368163.4047034,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368193.397779,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368193.406886,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368193.4069452,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368223.39839,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368223.4078932,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368223.4079487,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368253.3989491,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368253.408893,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368253.4089448,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368283.3993335,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368283.4150157,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368283.4150703,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368313.3999522,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368313.4092362,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368313.4092975,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368343.4005044,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368343.4113019,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368343.411358,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368373.4013097,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368373.410912,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368373.4109614,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368403.402181,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368403.4130418,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368403.413106,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368433.4029322,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368433.4128668,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368433.4129179,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368463.403225,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368463.4133303,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368463.413389,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368493.4035652,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368493.4141781,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368493.4142325,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368523.4038432,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368523.4188771,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368523.4189396,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368553.4043698,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368553.4141822,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368553.4142406,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368583.4051926,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368583.4153743,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368583.4154313,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368613.4055498,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368613.4147832,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368613.4148457,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368643.4062684,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368643.4177566,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368643.4178393,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368673.4073036,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368673.4168868,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368673.4169457,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368703.4083362,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368703.4181342,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368703.4181952,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368733.4089334,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368733.4187667,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368733.418842,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368763.4094834,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368763.4192924,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368763.4193482,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368793.409731,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368793.4193847,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368793.4194393,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368823.4098191,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368823.419875,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368823.4199288,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368853.409869,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368853.4199076,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368853.419959,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368883.4104614,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368883.421705,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368883.4217613,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368913.4105215,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785368913.4200788,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368913.4201386,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.63":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl":{}}} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 (EngineCore pid=74) INFO 07-29 23:35:15 [v1/worker/cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W729 23:35:15.766862786 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W729 23:35:15.766887154 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 74: [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [v1/worker/cpu_worker.py:109] OMP tid: 74, core 4 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [v1/worker/cpu_worker.py:109] OMP tid: 91, core 5 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [v1/worker/cpu_worker.py:109] OMP tid: 92, core 6 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [v1/worker/cpu_worker.py:109] OMP tid: 93, core 7 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [v1/worker/cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:15 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.63:51885 backend=gloo [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.63:51885 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:15 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:15 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors). [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:15 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading). [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:15 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:15 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:15 [config/compilation.py:1194] enabled custom ops: Counter() [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:15 [config/compilation.py:1195] disabled custom ops: Counter({'vocab_parallel_embedding': 1, 'logits_processor': 1}) [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:15 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ... [e2e-llm-inference-service] (EngineCore pid=74) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00 [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:24 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:34 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:44 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:53 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/86c9c3c579382eef68a98ac1d59b39811ba08abef3b4e90675a32c8dec3d7c90/rank_0_0/model [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:54 [compilation/monitor.py:76] Initial profiling/warmup run took 0.98 s [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:54 [v1/worker/cpu_model_runner.py:92] Warming up done. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:54 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 38.03 seconds [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:54 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode='hf' [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:54 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:54 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-29 23:35:54 [config/vllm.py:790] Asynchronous scheduling is disabled. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:54 [v1/engine/utils.py:1158] READY from local core engine process 0. [e2e-llm-inference-service] (EngineCore pid=74) WARNING 07-29 23:35:54 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:54 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-29 23:35:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:54 [entrypoints/openai/api_server.py:590] Supported tasks: ['generate'] [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/base.py:197] Warming up chat template processing... [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:141] There is no chat template fallback for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [renderers/hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:35:55 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /docs, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /redoc, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:35:55 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:36:03 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:36:03 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:36:03 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:36:03 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:36:03 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:36:03 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:36:03 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:36:03 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-29 23:36:03 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:36:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:36:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:36:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:36:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:36:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:36:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:37:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:37:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:37:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:37:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:37:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:37:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:38:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:38:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:38:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:38:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:38:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:38:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:39:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:39:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:39:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:39:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:39:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:39:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:40:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:40:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:40:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:40:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:40:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:40:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:41:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:41:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:41:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:41:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:41:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:41:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:42:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:42:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:42:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:42:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:42:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:42:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:43:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:43:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:43:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:43:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:43:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:43:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:44:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:44:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:44:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:44:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:44:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:44:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:45:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:45:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:45:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:45:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:45:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:45:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:46:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:46:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:46:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:46:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:46:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:46:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:47:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:47:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:47:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:47:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:47:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:47:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:48:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:48:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:48:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-29 23:48:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '03f06264-5d82-4736-abe0-d5726dbc6efe', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Wed, 29 Jul 2026 23:48:44 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785368040.3039339,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1785368040.3040545,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"e2e-test-llm-inference-service-48639af5","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3041797,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3048086,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3048387,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3051238,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785368040.325082,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3251488,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785368040.325256,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.325303,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3253517,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3254223,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3254533,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3255434,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3256638,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3256853,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3257506,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3257632,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3257723,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3258994,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3259106,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00066be80 DataConfig:{Sources:[{Plugin:0xc000816b40 Extractors:[0xc0004719c0]}]} FlowControlConfig: ParserRegistry:0xc0002f7340}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3436637,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3436947,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.34371,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437152,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437202,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437245,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437288,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437333,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437374,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437421,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437462,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437507,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437576,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3437624,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.343793,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3438008,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3438146,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3438334,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3438444,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3439403,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3439517,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.343978,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3443356,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3444364,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785368040.344531,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785368040.344716,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.344877,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3449965,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3450363,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3453867,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3454964,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3489244,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3489807,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.349451,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.3496616,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.4454625,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.4455137,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785368040.4455242,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.4455447,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785368040.4455833,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.4456017,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785368040.445741,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"54709cae-32b0-4a0c-831c-66ef492f3c26"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.5458345,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785368040.5458632,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785368054.4587479,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"e139f862-1f50-4b5a-8bea-64d68c5d4ef3"} [e2e-llm-inference-service] {"level":"info","ts":1785368163.19947,"caller":"inflightload/producer.go:308","msg":"Injected dynamic attribute into endpoint","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl-rank-0"} [e2e-llm-inference-service] {"level":"info","ts":1785368163.199534,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl","reconcileID":"c0a75ed6-7ab1-4fdf-8743-6b5a416af35a"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ba6aa3f4-e2c8-463e-b7c7-0d854288c7a2 [e2e-llm-inference-service] resourceVersion: '57849' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.50 [e2e-llm-inference-service] nodeName: ip-10-0-135-98.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv [e2e-llm-inference-service] uid: 7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 24ab0940-219b-4ed8-8976-17d62c46e72b [e2e-llm-inference-service] resourceVersion: '58828' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.63 [e2e-llm-inference-service] nodeName: ip-10-0-131-193.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl [e2e-llm-inference-service] uid: a503c2b0-6dfe-49fb-8089-5bd8d0b7f302 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a503c2b0-6dfe-49fb-8089-5bd8d0b7f302 [e2e-llm-inference-service] resourceVersion: '58826' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.63/23"],"mac_address":"0a:58:0a:85:00:3f","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.63/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.63\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:3f\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f [e2e-llm-inference-service] uid: 258f8eb1-5c6a-4d5d-920f-75e672d47d09 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-193 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"258f8eb1-5c6a-4d5d-920f-75e672d47d09"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:34:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.63"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-t96zt [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-t96zt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001030000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-t96zt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001030000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-131-193.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c32,c19 [e2e-llm-inference-service] fsGroup: 1001030000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdqb6 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-5lxhk [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:34:03Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:34:03Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] hostIP: 10.0.131.193 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.193 [e2e-llm-inference-service] podIP: 10.133.0.63 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.63 [e2e-llm-inference-service] startTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-29T23:34:03Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:4d57f8fe65a63b44f345487af2b831e64ddf7fb01126d4f51cc8cf49b7412d00 [e2e-llm-inference-service] containerID: cri-o://d2c172a4a5f357599999c550e72fee55324c6324e5b32f9cbba71b61ba135e9a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-t96zt [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001030000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001030000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-29T23:34:03Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://9b22d2a4d56aa1496299c4f533473547b5e862d42abb19e77732c80968269a11 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-t96zt [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001030000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001030000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bfq7xp [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 56094732-90fc-4ea7-b8cb-b551ca6cd72c [e2e-llm-inference-service] resourceVersion: '57155' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.53/23"],"mac_address":"0a:58:0a:84:00:35","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.53/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] uid: 3240bf87-4e36-46c9-8f74-f9385897d697 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-242 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3240bf87-4e36-46c9-8f74-f9385897d697"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-8n9r5 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-8n9r5 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001030000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-242.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c32,c19 [e2e-llm-inference-service] fsGroup: 1001030000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdqb6 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] hostIP: 10.0.128.242 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.242 [e2e-llm-inference-service] startTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-8n9r5 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b [e2e-llm-inference-service] resourceVersion: '57847' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.50/23"],"mac_address":"0a:58:0a:86:00:32","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.50/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.50\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:32\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b [e2e-llm-inference-service] uid: 5d4f63f1-e247-467c-9b0b-3e09749521a9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-98 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5d4f63f1-e247-467c-9b0b-3e09749521a9"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.50"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-lr9zk [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight:\ [e2e-llm-inference-service] \ 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-lr9zk [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001030000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-135-98.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c32,c19 [e2e-llm-inference-service] fsGroup: 1001030000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdqb6 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-w25b2 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] hostIP: 10.0.135.98 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.98 [e2e-llm-inference-service] podIP: 10.134.0.50 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.50 [e2e-llm-inference-service] startTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://49206a0fd920627fddb6f39e965dcbdc49b31ad51fedbc12de77766b3667b4ee [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-lr9zk [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001030000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001030000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2f39c4eb-7f75-4f5a-96b5-e40170bdde27 [e2e-llm-inference-service] resourceVersion: '57139' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-w25b2 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-w25b2"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-sdqb6"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-sdqb6 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-w25b2 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdqb6 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-w25b2 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 254754e7-897b-45c6-8667-94f54c02f1b0 [e2e-llm-inference-service] resourceVersion: '57104' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-5lxhk [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-5lxhk"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-sdqb6"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-sdqb6 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-5lxhk [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdqb6 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-5lxhk [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9c628f44-0264-4ee4-88c2-41cfb2e21f2c [e2e-llm-inference-service] resourceVersion: '57167' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.217.122 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.217.122 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 0b2e37df-b2e8-43e1-ae28-dfc964edc981 [e2e-llm-inference-service] resourceVersion: '57127' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.29.231 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.29.231 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: df627cd5-6437-47f5-b76a-11a59ddf8857 [e2e-llm-inference-service] resourceVersion: '58832' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 6a832db0-e2b2-4b88-8ca2-181b7751be1f [e2e-llm-inference-service] resourceVersion: '63662' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:44:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T23:44:00Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:44:00Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 66808f60-0910-45fc-b3a7-b45bd5784183 [e2e-llm-inference-service] resourceVersion: '57851' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n\ [e2e-llm-inference-service] \ plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 258f8eb1-5c6a-4d5d-920f-75e672d47d09 [e2e-llm-inference-service] resourceVersion: '58831' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: df627cd5-6437-47f5-b76a-11a59ddf8857 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"df627cd5-6437-47f5-b76a-11a59ddf8857"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 3240bf87-4e36-46c9-8f74-f9385897d697 [e2e-llm-inference-service] resourceVersion: '57152' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: 6a832db0-e2b2-4b88-8ca2-181b7751be1f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6a832db0-e2b2-4b88-8ca2-181b7751be1f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 5d4f63f1-e247-467c-9b0b-3e09749521a9 [e2e-llm-inference-service] resourceVersion: '57850' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: 66808f60-0910-45fc-b3a7-b45bd5784183 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"66808f60-0910-45fc-b3a7-b45bd5784183"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n\ [e2e-llm-inference-service] \ plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 7733cded-17b8-40fd-a824-adf2a6b1bb0b [e2e-llm-inference-service] resourceVersion: '57161' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 34eced9c-1b81-4f4d-95c8-c2676b9b7a5f [e2e-llm-inference-service] resourceVersion: '57112' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c133d3d9-f7e8-41ee-9b0c-82025ed2588b [e2e-llm-inference-service] resourceVersion: '57158' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d1a02587-5702-40f6-be88-33cccbae162a [e2e-llm-inference-service] resourceVersion: '57108' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-6fd6x [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 84c5de33-c8d5-496b-9c0e-422b7571324e [e2e-llm-inference-service] resourceVersion: '57848' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: 9c628f44-0264-4ee4-88c2-41cfb2e21f2c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:34:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9c628f44-0264-4ee4-88c2-41cfb2e21f2c"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.50 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv [e2e-llm-inference-service] uid: 7b0ebd49-e60f-4463-9b0d-2b74cbc7e69b [e2e-llm-inference-service] nodeName: ip-10-0-135-98.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-s5qgg2 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 07123d47-fa8c-4e71-8982-60f0655315ab [e2e-llm-inference-service] resourceVersion: '58830' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: 0b2e37df-b2e8-43e1-ae28-dfc964edc981 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:36:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0b2e37df-b2e8-43e1-ae28-dfc964edc981"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.63 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl [e2e-llm-inference-service] uid: a503c2b0-6dfe-49fb-8089-5bd8d0b7f302 [e2e-llm-inference-service] nodeName: ip-10-0-131-193.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 7733cded-17b8-40fd-a824-adf2a6b1bb0b [e2e-llm-inference-service] resourceVersion: '57161' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 34eced9c-1b81-4f4d-95c8-c2676b9b7a5f [e2e-llm-inference-service] resourceVersion: '57112' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c133d3d9-f7e8-41ee-9b0c-82025ed2588b [e2e-llm-inference-service] resourceVersion: '57158' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d1a02587-5702-40f6-be88-33cccbae162a [e2e-llm-inference-service] resourceVersion: '57108' [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T23:34:15Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57586' [e2e-llm-inference-service] uid: 59cc2d47-2dd3-42c9-93a5-80d305ff78a8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T23:34:15Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57586' [e2e-llm-inference-service] uid: 59cc2d47-2dd3-42c9-93a5-80d305ff78a8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57568' [e2e-llm-inference-service] uid: 58cf5409-ab64-4f52-829c-7e465c365946 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:14Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:01Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:01Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-29T23:34:03Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57363' [e2e-llm-inference-service] uid: 82db61d4-d235-441f-9199-e86fa4d4eee0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:02Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-29T23:34:03Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57205' [e2e-llm-inference-service] uid: db7de5b8-eb84-4938-a2a9-8269b22813d4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:26Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:26Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57764' [e2e-llm-inference-service] uid: 53ea1e57-65e0-4775-98d3-cc6cce299a1f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57244' [e2e-llm-inference-service] uid: e62adc52-53c6-43a3-b29b-2df38cd1c11e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57205' [e2e-llm-inference-service] uid: db7de5b8-eb84-4938-a2a9-8269b22813d4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:26Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:26Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57764' [e2e-llm-inference-service] uid: 53ea1e57-65e0-4775-98d3-cc6cce299a1f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57244' [e2e-llm-inference-service] uid: e62adc52-53c6-43a3-b29b-2df38cd1c11e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57205' [e2e-llm-inference-service] uid: db7de5b8-eb84-4938-a2a9-8269b22813d4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:26Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:26Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57764' [e2e-llm-inference-service] uid: 53ea1e57-65e0-4775-98d3-cc6cce299a1f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57244' [e2e-llm-inference-service] uid: e62adc52-53c6-43a3-b29b-2df38cd1c11e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb7aa9b8-b307-40b6-9558-51c672ec2f8f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-29T23:33:59Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: cb7aa9b8-b307-40b6-9558-51c672ec2f8f [e2e-llm-inference-service] resourceVersion: '57175' [e2e-llm-inference-service] uid: c92c6786-a2e7-4460-8cd1-22df88904b97 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:48:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] timestamp: '2026-07-29T23:48:27Z' [e2e-llm-inference-service] window: 13.689s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 98458397n [e2e-llm-inference-service] memory: 2400632Ki [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 16458797n [e2e-llm-inference-service] memory: 25324Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqdjqv [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-29T23:48:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] timestamp: '2026-07-29T23:48:21Z' [e2e-llm-inference-service] window: 17.439s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 56000573n [e2e-llm-inference-service] memory: 35304Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-29T23:48:45.890968] end - ❌ 901.805s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-29T23:36:04Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-29T23:34:36Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-29T23:34:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-48639af5 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:92: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] ERROR llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] ERROR llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] ERROR llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] ERROR llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 6 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] = 2 failed, 43 passed, 3 skipped, 25 warnings, 4 errors in 4286.13s (1:11:26) == [must-gather] [must-gather ] OUT 2026-07-30T00:13:33.742649489Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: f0d9d026-ee03-40d3-bb76-cba3903e5b76 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-30T00:13:33.781910217Z namespace/openshift-must-gather-bf526 created [must-gather] [must-gather ] OUT 2026-07-30T00:13:33.786273913Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-rh6jm created [must-gather] [must-gather ] OUT 2026-07-30T00:13:33.808711868Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:40.905335988Z [disk usage checker] Started [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:40.909239604Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.190847058Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.366010511Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.366039852Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.366039852Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.368919614Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.544233453Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.544264799Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.544264799Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.546591449Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.718885487Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.718917793Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.718917793Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.721291806Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.893796415Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.893919346Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.893919346Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:41.896724206Z Error getting logs from rhods-notebooks [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.069190338Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.069222830Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.069222830Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.071834100Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.239724959Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.239821328Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.239821328Z namespaces "istio-system" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.242314545Z Error getting logs from istio-system [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.411756464Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.411788823Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.411788823Z namespaces "knative-serving" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.414178166Z Error getting logs from knative-serving [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.576689888Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.576721252Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.576721252Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.579071134Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:42.873855449Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.178672614Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.328456535Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.411328793Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.413562103Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.551769195Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.636602915Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.639049613Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.776707848Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.866851470Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:43.869224422Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.006562726Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.090804207Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.093315781Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.237962763Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.326299817Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.328670048Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.470870745Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.553732539Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.555994423Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.701188974Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.791833328Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.794293424Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:44.933568470Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.019185916Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.021543801Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.164365154Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.252272082Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.255051081Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.397083125Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.485103238Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.487569356Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.627918258Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.712021640Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.714387030Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.856402908Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.913763106Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.939794818Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:45.942149800Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.084857471Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.170232599Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.172672066Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.313703926Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.402074000Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.404538410Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.545852600Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.629498764Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.631948157Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.769931097Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.856503439Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.859266805Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:46.997227487Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:47.080411201Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:47.082783670Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:47.225150584Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:47.313648392Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:47.316175349Z Error collecting info from [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:48.084914518Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:48.225654883Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:48.367797573Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:48.506410138Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:48.651678704Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:49.277979068Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:49.417942575Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:49.565256227Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:49.711974478Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:49.852872039Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:50.436571138Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:50.575357213Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:50.659106742Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:50.734467208Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:50.912757378Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:50.918683369Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:51.058926714Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:51.157189362Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:51.238122294Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:51.411964012Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:51.547105253Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:51.627608435Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:51.702571146Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:51.879745388Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.027033833Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.106838915Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.182589044Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.357556169Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.502858720Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.580858648Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.655154800Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.835785708Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:52.984738877Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:53.068307977Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:53.147175580Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:53.319023736Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:53.465692652Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:53.546573725Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:53.621060200Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:53.794763545Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:53.935921116Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.031601415Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.106663985Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.281017264Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.420862445Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.504703527Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.581437253Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.756554618Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.905622117Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:54.999219213Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.073653947Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.242203006Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.384762921Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.474779204Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.552335108Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.719218341Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.856280542Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.924032926Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:55.932867226Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.013761614Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.186775270Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.334242399Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.414005005Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.491587694Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.659168563Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.793505385Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.871797622Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:56.944409698Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:57.116700675Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:57.586768461Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:57.745612461Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:57.852526913Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:58.031809392Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:58.479763737Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:58.653943596Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:58.770610940Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:58.948278645Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:59.395319032Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:59.551825462Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:59.658010152Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:13:59.830065987Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:00.276609203Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:00.431653302Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:00.550380620Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:00.729818328Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:00.928730732Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:01.282464966Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:01.469165919Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:01.597783633Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:01.779175487Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:02.509297511Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:02.876657400Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:03.282960031Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:03.462886984Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:04.073229911Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:04.255195902Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:04.379917273Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:04.552885702Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:05.086643514Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:05.272090516Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:05.394624142Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:05.576231412Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:05.934707586Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:06.126066412Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:06.315839429Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:06.439437510Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:06.616530156Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:07.148868712Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:07.324402310Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:07.447800842Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:07.615016145Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:08.142161638Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:08.321909319Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:08.446641237Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:08.615878134Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:09.149631102Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:09.328096967Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:09.454726336Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:09.625099169Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:10.164537310Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:10.341030946Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:10.463000105Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:10.634983523Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:10.940705495Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:11.185105828Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:11.365924200Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:11.495025610Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:11.666500350Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:12.227351195Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:12.410532553Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:12.529946836Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:12.703034152Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:13.236568891Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:13.417864736Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:13.544248583Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:13.718783995Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:14.277063341Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:14.463899440Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:14.586122222Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:14.762759576Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:15.303180043Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:15.489768416Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:15.617007843Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:15.789713534Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:15.945569296Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:16.342919309Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:16.520951996Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:16.640285829Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:16.812810339Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:17.358387610Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:17.541637810Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:17.660156574Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:17.829002771Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:18.364528969Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:18.570755905Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:18.713762261Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:18.885710250Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:19.430537985Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:19.608750335Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:19.727142120Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:19.900923812Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:20.433698461Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:20.616973872Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:20.739429655Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:20.917398059Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:20.950175172Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:21.454936050Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:21.639818778Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:21.769869342Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:21.952773387Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:22.515646285Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:22.703289008Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:22.833513234Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:23.012193904Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:23.566341975Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:23.748441735Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:23.872644510Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:24.045856990Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:24.603562658Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:24.784460869Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:24.908816986Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:25.080626142Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:25.628340462Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:25.806595818Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:25.927326969Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:25.955038914Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:26.100811918Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:26.639780424Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:26.823017127Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:26.944715087Z Wrote inspect data to must-gather. [must-gather] [must-gather-877bs] POD 2026-07-30T00:14:26.967436414Z Caches written to disk [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:27.975581463Z waiting for gather to complete [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:27.97789574Z downloading gather output [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.241490814Z receiving incremental file list [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.252081611Z ./ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.252208744Z aggregated-discovery-api.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.252608516Z aggregated-discovery-apis.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.253491801Z event-filter.html [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.253880483Z timestamp [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.254035587Z version [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.259979019Z cluster-scoped-resources/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26000819Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26001891Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260157364Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26035816Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260378971Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260500444Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260596227Z namespaces/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260604057Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260655679Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260732241Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260802293Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260871295Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.260935777Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26103757Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261132122Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261230375Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261284757Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261351609Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261406701Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261461522Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261555565Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261616916Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261669478Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261769341Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261807542Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.261879254Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262024908Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262117811Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262214694Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262306176Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26241063Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262548604Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262732109Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262795091Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262853923Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262934335Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.262991907Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263050598Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.2631004Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263203342Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263261744Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263320096Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263376087Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263432219Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263488491Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263543892Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263599984Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263657116Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263713187Z namespaces/kuadrant-system/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263775289Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263843991Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263916683Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.263984265Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.264040127Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.264134749Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26448479Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26483553Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.264905972Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.264966433Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265031526Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265082287Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26518197Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265231941Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265294373Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265387476Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265447967Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265497829Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265695415Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.265833849Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.266375464Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.266462687Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.266880919Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.267018883Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26724821Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.267392994Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.267449336Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.2675838Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.267637021Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.267699743Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.267753154Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.267850397Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26793959Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.267986981Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.268042993Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.268085324Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.268130795Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.268137856Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.268183907Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino-7cf4765b85-gvscv.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.268269629Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26828151Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.26828701Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.268311561Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.279949807Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.28002853Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280062131Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280139023Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/authorino-operator-56fb6f6856-b9spq.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280233816Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280245356Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280250606Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280288377Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280495213Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280573276Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280616807Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280669648Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/dns-operator-controller-manager-ccbdf686-dccw7.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280787792Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280799992Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280806372Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.280838663Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281003668Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.28108499Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281124681Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281183033Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin-598f86468b-vrnns.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281264256Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281271536Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281275666Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281326027Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.28141687Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281501252Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281537953Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281607606Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/kuadrant-operator-controller-manager-66866f5fcc-tpm8g.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281733179Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281740639Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.281744659Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.28178364Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293293104Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293379856Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293433538Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.29348653Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador-limitador-85d9db4d5d-rn5v2.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293623743Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293631174Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293635174Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293681205Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293788148Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293883081Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293916582Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.293965233Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/limitador-operator-controller-manager-774d94476f-48g2h.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294073907Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294082087Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294086047Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294140429Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294285073Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294355505Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294399296Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294425947Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294487148Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.29452026Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294575971Z namespaces/openshift-ingress/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294641303Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294716405Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294756566Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294799517Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294845299Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.294950522Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.29522509Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295481277Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295530709Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295559519Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295634122Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295687123Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295758745Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295806147Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295848528Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295950591Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.295993822Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.296032233Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.296488546Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.2966044Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.296849017Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.2969587Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297190197Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297333561Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297591738Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297675131Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297708032Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297788794Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297828695Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297884697Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.297950219Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298030431Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298047492Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298111073Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298167515Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298184636Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298242217Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298253088Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298293619Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/istiod-openshift-gateway-5677ccb49b-dz27l.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298406852Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298416732Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298421722Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.298449393Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.313392196Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.313460628Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.313506489Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.31354251Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.314189029Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.31420714Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.3142139Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.31422901Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.315787565Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.315891118Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.31595171Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router-default-6d7fb7b5bf-sg26j.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316059183Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316067104Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316071184Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316108565Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/current.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316223508Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/previous.insecure.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.31629919Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/previous.log [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316337191Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316376232Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316431554Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.316477185Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.322283704Z [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.322307594Z sent 2,844 bytes received 985,382 bytes 658,817.33 bytes/sec [must-gather] [must-gather-877bs] OUT 2026-07-30T00:14:28.322320305Z total size is 14,267,748 speedup is 14.44 [must-gather] [must-gather ] OUT 2026-07-30T00:14:28.374096494Z namespace/openshift-must-gather-bf526 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: f0d9d026-ee03-40d3-bb76-cba3903e5b76 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-30T00:14:28.49345131Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: f0d9d026-ee03-40d3-bb76-cba3903e5b76 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-30T00:14:28.521280676Z namespace/openshift-must-gather-6zt68 created [must-gather] [must-gather ] OUT 2026-07-30T00:14:28.530322718Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-jbm4w created [must-gather] [must-gather ] OUT 2026-07-30T00:14:28.573361055Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:29.956739817Z [disk usage checker] Started [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:29.960403958Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:30.201110899Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:30.366339774Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:30.456931999Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:30.554942711Z Gathering data for ns/default... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:30.905663287Z Gathering data for ns/openshift... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:31.353326475Z Gathering data for ns/kube-system... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:32.093606652Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:32.456986972Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:32.457180505Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:32.457180505Z namespaces "assisted-installer" not found [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.028164631Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.034050496Z INFO: Gathering HAProxy config files [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.034713729Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.042685617Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.048597381Z Executing Istio gather script [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.054880372Z INFO: Collecting host service logs for crio [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.055218668Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.055555507Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.055590245Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.055977750Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.056318706Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.056645990Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.057002161Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.057367005Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.057698796Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.058066659Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.058420887Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.059492214Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.159395646Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.402658471Z No resources found [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.613652694Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.613746676Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.662630497Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.867646388Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:33.886382961Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:34.016486743Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:34.193435267Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:34.359901158Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:34.569883548Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:34.748598376Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:34.881780193Z No resources found [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:34.989436075Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:34.999763604Z tar: Removing leading `/' from member names [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.008877274Z INFO: HAProxy config collection complete. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.071971443Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.075664580Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.131172907Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.177012465Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.216885481Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-dd689 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.216885481Z insights-runtime-extractor-5d2kg [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.216885481Z insights-runtime-extractor-8rxsh [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.216885481Z insights-runtime-extractor-rr5g7 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.438720502Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.455972499Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.660923709Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.712405152Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.719774777Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.808713232Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:35.962251706Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.018102654Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.018102654Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.074826913Z INFO: 'previous-logs' folder not found on ip-10-0-128-242.ec2.internal, skipping... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.295999248Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.321216574Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.430704122Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.513939273Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.751606146Z INFO: 'previous-logs' folder not found on ip-10-0-131-193.ec2.internal, skipping... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.823627066Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.884538778Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.940204537Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:36.944307102Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.069181648Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.153330395Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.329738628Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.392376202Z INFO: 'previous-logs' folder not found on ip-10-0-135-98.ec2.internal, skipping... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.392419682Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.392459133Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.397779763Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.443652113Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.448057575Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.670050576Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.910692080Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.922746414Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:37.926018193Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.011716012Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.077029432Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.142915282Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.266309043Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.383769149Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.387460939Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.425378630Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.494381262Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.530890665Z INFO: INTERCONNECT MODE [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.530931056Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.648233932Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.722416270Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.758021814Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.764047500Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.811583340Z INFO: Gathering OVN_Northbound from ovnkube-node-grsjn... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.811918404Z INFO: Gathering OVN_Northbound from ovnkube-node-gzpfl... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.812281915Z INFO: Gathering OVN_Northbound from ovnkube-node-jvw95... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.812529681Z INFO: Gathering OVN_Southbound from ovnkube-node-grsjn... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.812804135Z INFO: Gathering OVN_Southbound from ovnkube-node-gzpfl... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.813097586Z INFO: Gathering OVN_Southbound from ovnkube-node-jvw95... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:38.832799604Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.144993486Z tar: Removing leading `/' from member names [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.162298806Z tar: Removing leading `/' from member names [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.174440905Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.181172053Z tar: Removing leading `/' from member names [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.247023726Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.265366184Z tar: Removing leading `/' from member names [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.291663298Z tar: Removing leading `/' from member names [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.303182210Z tar: Removing leading `/' from member names [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.391348153Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.572838872Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.918186403Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.931609278Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.934919372Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.935088667Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.935739848Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.937586873Z 6.7M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.940492794Z ovnk_database_store/ [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.940492794Z ovnk_database_store/ovnkube-node-jvw95_sbdb [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:39.983893284Z ovnk_database_store/ovnkube-node-gzpfl_sbdb [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.022405365Z ovnk_database_store/ovnkube-node-grsjn_sbdb [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.067357402Z ovnk_database_store/ovnkube-node-grsjn_nbdb [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.084937095Z ovnk_database_store/ovnkube-node-jvw95_nbdb [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.097316367Z ovnk_database_store/ovnkube-node-gzpfl_nbdb [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.116030637Z INFO: Network log collection complete. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.271718944Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.280744846Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.395902565Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.554680136Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:40.557908957Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.089647564Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.485574172Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.557026279Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.643834212Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.718376119Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.813838586Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.972436053Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.976895012Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:41.988073839Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:42.062686340Z Gathering data for ns/openshift-console... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:42.268025506Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:42.520510197Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:42.524154666Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:42.672112155Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:42.740303512Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:42.980233901Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:42.982676416Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.159825235Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.234137619Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.305020151Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.465745746Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.469524878Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.655044128Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.689101594Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.899443105Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:43.902648021Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.107598638Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.322075991Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.357720628Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.361135634Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.364715848Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.576300251Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.794754390Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.797044295Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:44.964716893Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.055332552Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.292972516Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.296574558Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.386123900Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.401675786Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.489037943Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.519049724Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.763072777Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.765428864Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:45.966243449Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.206720823Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.210530197Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.249627421Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.422956528Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.653699783Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.668879478Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.671520226Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.741199868Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:46.858421469Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.089200850Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.093840951Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.280846310Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.526743358Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.530562217Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.725417247Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.806017838Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.939817378Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:47.942377651Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:48.131799489Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:48.345472501Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:48.348834012Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:48.555036480Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:48.768088980Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:48.771246432Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:48.967329524Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:48.978611983Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:49.198769649Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:49.201822294Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:49.412774307Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:49.648849862Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:49.651369167Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:49.819358762Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:49.881126655Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.063965593Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.066766431Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.135041657Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.284132284Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.413161836Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.475140718Z Collecting performance related data for node ip-10-0-131-193.ec2.internal [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.482774908Z Collecting performance related data for node ip-10-0-135-98.ec2.internal [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.491916331Z Collecting performance related data for node ip-10-0-128-242.ec2.internal [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.544980265Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.551453178Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.587530452Z Gathering data for ns/kserve... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:50.805188690Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:51.103355700Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:51.282263737Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:51.576070194Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:52.219187743Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:52.434270779Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:52.578341448Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:52.708706248Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:52.916229768Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:53.331503049Z Gathering data for ns/openshift-config... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:53.619469356Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:53.755392180Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:53.851234023Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:54.028182298Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:54.324763330Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:54.588499717Z Done executing Istio gather script [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:54.684603907Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:54.989377095Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:55.368598680Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:55.421528598Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:55.721983564Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:56.043775026Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:56.417057699Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:56.784548277Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:57.140677276Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:57.711156767Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:59.302359383Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:59.684908165Z Collecting kubelet logs for node ip-10-0-131-193.ec2.internal [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:59.686870244Z Collecting kubelet logs for node ip-10-0-135-98.ec2.internal [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:14:59.689117499Z Collecting kubelet logs for node ip-10-0-128-242.ec2.internal [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:00.123111751Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-6zt68 namespace [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:00.127845182Z INFO: Node performance data collection complete. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:00.429766213Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:00.676885129Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:01.019819428Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:01.582628316Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:02.031214805Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:02.474301739Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:02.819809992Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:03.349471353Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:03.699481658Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:04.159534273Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:04.469828365Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:04.794281958Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:05.437616508Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:05.478341384Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:05.820722860Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:06.158057380Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:06.566377264Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:07.585918709Z Wrote inspect data to must-gather. [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:07.585965526Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:07.585965526Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:07.735827118Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-qz5r8] POD 2026-07-30T00:15:08.570321442Z Caches written to disk [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.581916932Z waiting for gather to complete [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.584558728Z downloading gather output [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.82256085Z receiving incremental file list [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.841004444Z ./ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.841082806Z aggregated-discovery-api.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.841478627Z aggregated-discovery-apis.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.842633251Z event-filter.html [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.844970278Z timestamp [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.845065831Z version [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.853639559Z cluster-scoped-resources/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.85366229Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.853697541Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.853713091Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.853847945Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854003889Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854079262Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854132173Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854226566Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854307238Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854421572Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854467373Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854604537Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.854670209Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.855141542Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.855181514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.855354868Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.855573545Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.85576229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.855913835Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.859309303Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.8609244Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.861044103Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.861162047Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.86127294Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.861404463Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.861878097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.862042072Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.862189456Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.86230354Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.863112193Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.86334415Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.863465353Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.863626838Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.864813312Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.86506676Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.865235514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.865391379Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.865697568Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.865832751Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.86611303Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.866368727Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.867056967Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.8671699Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.867294184Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.867411897Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.8675129Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.867622823Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.867780308Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.867983614Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.868772127Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.868920771Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.869052835Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.87096012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterservingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.871527906Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.871683621Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.871875626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.872099633Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.87236074Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.872797733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.872936107Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.87304575Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.873170934Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.873290857Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.87340205Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.873511794Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.873671558Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.873827793Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.873970597Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.874226554Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.874359978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.874468061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.87511742Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.875257794Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.875371677Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.875555133Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.876316335Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.876432298Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.876620613Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.876731167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.876925622Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.877058486Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.87720501Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.877337854Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.877468268Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.877584291Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.877721595Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.877848839Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.877987513Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.878135397Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.878270341Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.878394245Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.879125106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.879572329Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.879786745Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.880255388Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.880373152Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.881185695Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.881316739Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.881433393Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.881542876Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.881665269Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.881849925Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.882136113Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.882266577Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.88238309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.882539634Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.882667188Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.882782061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.882947336Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.88308012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.887630122Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.888088885Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.888658122Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.888851457Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.889026432Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.889161156Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.88929866Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.889408303Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.889514116Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.890355081Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.892240395Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.895369696Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.895506929Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.895640064Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.895764487Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.895937682Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.895988063Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.896146978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.907388643Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.907668671Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.913030336Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.918145174Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.918293219Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.918572067Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.918722061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.918881045Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.919005079Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.919154804Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.919420081Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.919531644Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.919638707Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.919829033Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.919980497Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.920102261Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.920217154Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.920328077Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.920466531Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.920617056Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.920733259Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.920890174Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.921015967Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.921152011Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.921247024Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.921382558Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.921494421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.921614004Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.921918113Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.922047247Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.922300084Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.922414088Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.922514121Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.922632484Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.924673963Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.924818427Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.924946751Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.925059974Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.925159697Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.925310431Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.925430335Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.925579189Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.925706703Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.927264398Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.927427803Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.927545236Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.92769118Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.927805574Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.928002579Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.928249696Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.928752491Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.929040029Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.929160643Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.929268696Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.929375849Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.930044628Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.930232354Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.930349237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.93184674Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.932018495Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.93216899Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.932275403Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.932442788Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.932710185Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.93287598Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.933042225Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.933442577Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.93355506Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.933664683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.933829568Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.933996502Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.934131606Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.934271921Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.934428745Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.934717613Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.934768535Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.934878398Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.934969981Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935050323Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935141966Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.93530197Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935408533Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935490686Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935581058Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935674201Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935766354Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935847796Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.935957959Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936045852Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936139404Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936234657Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.9363278Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936422463Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936507235Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936594038Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.93668438Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936789913Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936875066Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.936984759Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937071442Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937161284Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937253337Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937338949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937429832Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937522035Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937606197Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.93769706Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937790932Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.937877055Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938011509Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938094071Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938182943Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938278726Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938379129Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938464142Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938555844Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938641017Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938731939Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938846223Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.938956046Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939040578Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939134911Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939249705Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939341097Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.93943928Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939523692Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939619255Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939717578Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939805441Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.939919924Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940015967Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940109399Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940198372Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940293045Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940381097Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.9404754Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940573133Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940659825Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940767248Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940885992Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.940996235Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941084707Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941192301Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941282003Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941378656Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941468069Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941563531Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941652044Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941746937Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941839429Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.941957803Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942046865Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942145988Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942234191Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942333174Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942428356Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942521529Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942618082Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942707785Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942817808Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.942922821Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943002043Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943096436Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943192579Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943288001Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943376654Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943491087Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.94357852Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943676253Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943762465Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943855678Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.943977681Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944064464Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944166587Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944255289Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944353632Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944441295Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944537497Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.94462514Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944718543Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944833786Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944910668Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944923829Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.944965Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-7rhw5.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.945078193Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-f55sv.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.945166716Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-plvgb.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.945369102Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.945423153Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.945576247Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.945705351Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.945797404Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.945919637Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.946334429Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.946436802Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.946521875Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.946612257Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.946720721Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.946801703Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.946920676Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947004199Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947100252Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947187924Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947287547Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947372509Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947466502Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947561335Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947644497Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.94773267Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947823522Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.947932276Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948011958Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948051329Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948164882Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948254945Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948376848Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948461801Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948569774Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948671997Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948840202Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.948954885Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949051708Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949156041Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949248114Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949390118Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.94947921Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949583413Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949673056Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949772539Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949916273Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.949997775Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950088568Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950191691Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950263193Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950309364Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950400727Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950439818Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950522191Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950550701Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950645564Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950675035Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950742237Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950783698Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950878271Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950928132Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950975933Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.950983914Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951030005Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951124528Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951227201Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951278042Z cluster-scoped-resources/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951294533Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951342064Z cluster-scoped-resources/core/nodes/ip-10-0-128-242.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951508559Z cluster-scoped-resources/core/nodes/ip-10-0-131-193.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951651563Z cluster-scoped-resources/core/nodes/ip-10-0-135-98.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951762656Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951804717Z cluster-scoped-resources/core/persistentvolumes/pvc-c2043d33-7945-4454-be43-31b1a1e806e0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.95188005Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951959492Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.951997763Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.952105866Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.952190909Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.952293292Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.952381044Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.952465027Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.952612241Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.952707564Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.952819107Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.95292988Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953024903Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953116795Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953210458Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953307491Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953411884Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953501566Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953595879Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953681152Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953769084Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.953880277Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.95397757Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954041472Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954083403Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954168976Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954250568Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954334681Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954418993Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954520146Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954678881Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.954808274Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.95499706Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955056622Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955066252Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955111653Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955184105Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955193095Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955224807Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.95533221Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.9553641Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955432113Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955441393Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955489954Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955581967Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955725101Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955777112Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955786293Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955824064Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955983018Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.955996799Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956019839Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956205665Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956240856Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956364859Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956412171Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956499493Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956541145Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956604986Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956656958Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956779431Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956820122Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956892895Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.956958397Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957024139Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.95707005Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957141202Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957189403Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957267555Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957339508Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.9574119Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957454321Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957507542Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957538243Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957602285Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957641026Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957699698Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957743189Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957845682Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957853123Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.957928925Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958011837Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958085509Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958189862Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958293545Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.95844418Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958548883Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958658316Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958766619Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958856792Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.958961724Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959005896Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959103629Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959184631Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959333795Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959461409Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959551372Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959646154Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959731357Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959819459Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.959940873Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960024295Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960112698Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.96020668Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960287693Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960380745Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960473838Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960561111Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960648693Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960749996Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960848659Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.960962852Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.961072085Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.961100866Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.9612225Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.961297372Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.961387255Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.961535219Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.961631812Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.961716234Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.961814787Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.9619247Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962014183Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962100355Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962189518Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962283571Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962365753Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962461736Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962548648Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962641851Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962733293Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.962824156Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.96296012Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963053513Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963152996Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963237188Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963325911Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963420504Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963524156Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963609659Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963705382Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963792504Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963916448Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963959129Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.963966819Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964025711Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964062492Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964073382Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964119364Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964190536Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964198006Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964254008Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964315529Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964361121Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-128-242.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964451553Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-193.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964531036Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-135-98.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964596267Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964649409Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964735942Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964812074Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964854805Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-432cb4c176574272915f299d2e42849927080919195d4868bd96f29d960cb449.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964936197Z host_service_logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.964990759Z host_service_logs/masters/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.96503236Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965109672Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965170984Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965243546Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.96538604Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965502454Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965605687Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965676149Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965751081Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965819183Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965923666Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965946406Z ingress_controllers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965957347Z ingress_controllers/default/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.965962417Z ingress_controllers/default/router-default-6d7fb7b5bf-sg26j/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.966005018Z ingress_controllers/default/router-default-6d7fb7b5bf-sg26j/haproxy.config [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.966111681Z insights-data/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.966119331Z istio/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.966155052Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.966264006Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.966841462Z istio/event-filter.html [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967027678Z istio/timestamp [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967125701Z istio/cluster-scoped-resources/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967137751Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967141881Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967164922Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967262344Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967290885Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967383598Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967411369Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.96746903Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.967666146Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.968576593Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.96884268Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.969058516Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.96951231Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.969723636Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.970256141Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.971195368Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.971375574Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.971538638Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.971694683Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.971835277Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.972035722Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.972312781Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.972507696Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.972783584Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97298708Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.973376131Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.973504815Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.973640549Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.973799073Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.973806554Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.973889716Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.973959558Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.973967889Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97401908Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974115823Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974257787Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974331119Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97438525Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974489133Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974669179Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97472576Z istio/namespaces/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97473304Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974736991Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/gateway.networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974743301Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974802843Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/gateway.networking.k8s.io/httproutes/autoscale-hpa-deploy-kserve-route.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974887405Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/inference.networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974903596Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.974962367Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/inference.networking.k8s.io/inferencepools/autoscale-hpa-deploy-inference-pool.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975023749Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/inference.networking.x-k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975033319Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975088691Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/inference.networking.x-k8s.io/inferencepools/autoscale-hpa-deploy-inference-pool.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975155203Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/networking.istio.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975165753Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/networking.istio.io/destinationrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975209904Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/networking.istio.io/destinationrules/autoscale-hpa-deploy-kserve-scheduler.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975304847Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/networking.istio.io/destinationrules/autoscale-hpa-deploy-kserve-shadow-svc.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97538783Z istio/namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/networking.istio.io/destinationrules/autoscale-hpa-deploy-kserve-workload-svc.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975508583Z istio/namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975516393Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975520953Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975562875Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975659267Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975666698Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975715399Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975775011Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975783151Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975816992Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975926215Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975944836Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.975964236Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976059599Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976211923Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976328507Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976377298Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97644183Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976485031Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976561603Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976602974Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976691667Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.976967945Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977167821Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977210912Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977246773Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977323975Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977361496Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977455149Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977513761Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977567102Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977644835Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977722657Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.977758728Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.978215351Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.978315954Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97852719Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.978609552Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97884707Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979000424Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.97921315Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979319353Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979355944Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979435496Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979443097Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979495918Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979578051Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979611532Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979681663Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979726895Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.979812617Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980001013Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980057274Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980109366Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980174658Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980184448Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.98023749Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980340333Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980491827Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980597Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.98061609Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980708113Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980719493Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980729424Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/istiod-openshift-gateway-5677ccb49b-dz27l.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980894789Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980914519Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.980919539Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.98094704Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.989532659Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.98960545Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.989624721Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.989685243Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.989824207Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.989833297Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.989838497Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.989848968Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99095894Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991037952Z istio/namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991084713Z istio/namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router-default-6d7fb7b5bf-sg26j.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991196467Z istio/namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991212247Z istio/namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991217227Z istio/namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991225037Z istio/namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991345981Z istio/namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991419443Z istio/namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991477385Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991493465Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991580218Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991615019Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99166823Z machine_config_ondisk/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99167725Z machine_config_termination_logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99168223Z monitoring/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991687131Z monitoring/alertmanager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991704441Z monitoring/alertmanager/status.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991805694Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991889347Z monitoring/prometheus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.991921188Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.992045101Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.992108693Z monitoring/prometheus/rules.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.992750092Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.992814893Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.992882725Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.993805642Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.993818732Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.993871034Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.993985467Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.994047179Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.994143012Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.994154392Z monitoring/prometheus/status/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.994210434Z monitoring/prometheus/status/config.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.994657927Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.994723098Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.994820121Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995091439Z namespaces/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995102589Z namespaces/cert-manager-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99510697Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99511092Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995141151Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995226863Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995239653Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995291855Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995510521Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995724038Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.995882122Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99616791Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.996410237Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.996446878Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-zdbwj.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.996605723Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.996712226Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.996789148Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99683993Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.996945053Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.996977284Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.997048906Z namespaces/cert-manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.997056596Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.997063336Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.997118018Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.997285463Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.997458138Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.997596832Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.997795337Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998064685Z namespaces/default/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998108976Z namespaces/default/default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998181509Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99822292Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998255971Z namespaces/default/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998314742Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998391555Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998479077Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998563189Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998609241Z namespaces/default/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998629742Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998698924Z namespaces/default/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998745985Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998827577Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998880629Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.998940271Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999015583Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999073054Z namespaces/default/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999108335Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999226309Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999312301Z namespaces/default/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999487616Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999566639Z namespaces/default/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999647441Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.999786325Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:09.99996895Z namespaces/default/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000049783Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000088994Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000162486Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000198687Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000263329Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.0002993Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000387282Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000464695Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000522856Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000558997Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000612819Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.00064058Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000710372Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000724332Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000792154Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.000962419Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001150034Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001286108Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001480834Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001660859Z namespaces/default/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.00170344Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001753522Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001793923Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001845864Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001853245Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001857445Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.001923317Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.002164104Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.002349279Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.002484083Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.002682769Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.002884005Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.002898505Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.002902805Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.002969017Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.00306804Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003116551Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003149692Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003242035Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003301157Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003311227Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003376329Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003525533Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003714778Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.003852553Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.004093329Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.004279145Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.004288045Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.004307286Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.004350437Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.004503261Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.004689007Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.004835191Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.005059168Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.005253053Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.005265143Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.005277054Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.005321875Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.005467349Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.005646794Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.005789478Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006008855Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006206471Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006217891Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006225991Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006273312Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006432757Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006607402Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006749536Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.006964412Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007147418Z namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007157218Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007162138Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007175589Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007261861Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007321273Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007369784Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007452157Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007500588Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007518188Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.00758417Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.007716194Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.00791281Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.008060674Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.00825886Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.008437595Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.008445455Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.008452055Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.008508977Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.008663272Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.008854307Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009012212Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009224078Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009447804Z namespaces/kserve/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009491986Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009567888Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009597439Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009678361Z namespaces/kserve/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009707022Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.009803375Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010044362Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010237587Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010267028Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.01032341Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010380931Z namespaces/kserve/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010436023Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010523586Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010620678Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.01066061Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010784003Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010830144Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010839045Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010887136Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.010999769Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.011076442Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.011178954Z namespaces/kserve/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.011221316Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.011455102Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.011566496Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.011984788Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.012078671Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.012351438Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.012480722Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.01275625Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.012872263Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.012927165Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013018008Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013070119Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.01311387Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013176282Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013255725Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013341897Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013380198Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.01343354Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013508622Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013553443Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013646076Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013702567Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013754629Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.0137959Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013822741Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.013876172Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014045117Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014240463Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014378457Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014571603Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014792169Z namespaces/kserve/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014799619Z namespaces/kserve/pods/kserve-controller-manager-844cf74686-w286v/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014851471Z namespaces/kserve/pods/kserve-controller-manager-844cf74686-w286v/kserve-controller-manager-844cf74686-w286v.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014963554Z namespaces/kserve/pods/kserve-controller-manager-844cf74686-w286v/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014975714Z namespaces/kserve/pods/kserve-controller-manager-844cf74686-w286v/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.014979694Z namespaces/kserve/pods/kserve-controller-manager-844cf74686-w286v/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015023736Z namespaces/kserve/pods/kserve-controller-manager-844cf74686-w286v/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015150159Z namespaces/kserve/pods/kserve-controller-manager-844cf74686-w286v/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015217421Z namespaces/kserve/pods/kserve-controller-manager-844cf74686-w286v/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015275083Z namespaces/kserve/pods/llmisvc-controller-manager-7586c4769-btsn4/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015309764Z namespaces/kserve/pods/llmisvc-controller-manager-7586c4769-btsn4/llmisvc-controller-manager-7586c4769-btsn4.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015407517Z namespaces/kserve/pods/llmisvc-controller-manager-7586c4769-btsn4/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015416407Z namespaces/kserve/pods/llmisvc-controller-manager-7586c4769-btsn4/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015422117Z namespaces/kserve/pods/llmisvc-controller-manager-7586c4769-btsn4/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.015460788Z namespaces/kserve/pods/llmisvc-controller-manager-7586c4769-btsn4/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.016995833Z namespaces/kserve/pods/llmisvc-controller-manager-7586c4769-btsn4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017060735Z namespaces/kserve/pods/llmisvc-controller-manager-7586c4769-btsn4/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017116276Z namespaces/kserve/pods/model-serving-api-7b954ffc95-6t2tn/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017152717Z namespaces/kserve/pods/model-serving-api-7b954ffc95-6t2tn/model-serving-api-7b954ffc95-6t2tn.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017279301Z namespaces/kserve/pods/model-serving-api-7b954ffc95-6t2tn/server/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017288291Z namespaces/kserve/pods/model-serving-api-7b954ffc95-6t2tn/server/server/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017292531Z namespaces/kserve/pods/model-serving-api-7b954ffc95-6t2tn/server/server/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017300831Z namespaces/kserve/pods/model-serving-api-7b954ffc95-6t2tn/server/server/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017403265Z namespaces/kserve/pods/model-serving-api-7b954ffc95-6t2tn/server/server/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017469586Z namespaces/kserve/pods/model-serving-api-7b954ffc95-6t2tn/server/server/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017517858Z namespaces/kserve/pods/odh-model-controller-8588677cfb-n4kbt/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017561859Z namespaces/kserve/pods/odh-model-controller-8588677cfb-n4kbt/odh-model-controller-8588677cfb-n4kbt.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017648772Z namespaces/kserve/pods/odh-model-controller-8588677cfb-n4kbt/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017656342Z namespaces/kserve/pods/odh-model-controller-8588677cfb-n4kbt/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017666872Z namespaces/kserve/pods/odh-model-controller-8588677cfb-n4kbt/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.017721134Z namespaces/kserve/pods/odh-model-controller-8588677cfb-n4kbt/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045058284Z namespaces/kserve/pods/odh-model-controller-8588677cfb-n4kbt/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045107846Z namespaces/kserve/pods/odh-model-controller-8588677cfb-n4kbt/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045140587Z namespaces/kserve/pods/s3-init-mhzx7/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045199528Z namespaces/kserve/pods/s3-init-mhzx7/s3-init-mhzx7.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045314441Z namespaces/kserve/pods/s3-init-mhzx7/download-hf-model/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045323772Z namespaces/kserve/pods/s3-init-mhzx7/download-hf-model/download-hf-model/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045328952Z namespaces/kserve/pods/s3-init-mhzx7/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045339762Z namespaces/kserve/pods/s3-init-mhzx7/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045453705Z namespaces/kserve/pods/s3-init-mhzx7/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045523518Z namespaces/kserve/pods/s3-init-mhzx7/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045539558Z namespaces/kserve/pods/s3-init-mhzx7/s3-init/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045547728Z namespaces/kserve/pods/s3-init-mhzx7/s3-init/s3-init/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045552398Z namespaces/kserve/pods/s3-init-mhzx7/s3-init/s3-init/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.04561684Z namespaces/kserve/pods/s3-init-mhzx7/s3-init/s3-init/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045887268Z namespaces/kserve/pods/s3-init-mhzx7/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.0459724Z namespaces/kserve/pods/s3-init-mhzx7/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.045993301Z namespaces/kserve/pods/seaweedfs-64568bcd49-qvr5s/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046048963Z namespaces/kserve/pods/seaweedfs-64568bcd49-qvr5s/seaweedfs-64568bcd49-qvr5s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046134265Z namespaces/kserve/pods/seaweedfs-64568bcd49-qvr5s/seaweedfs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046141675Z namespaces/kserve/pods/seaweedfs-64568bcd49-qvr5s/seaweedfs/seaweedfs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046152136Z namespaces/kserve/pods/seaweedfs-64568bcd49-qvr5s/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046200657Z namespaces/kserve/pods/seaweedfs-64568bcd49-qvr5s/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046387983Z namespaces/kserve/pods/seaweedfs-64568bcd49-qvr5s/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046457625Z namespaces/kserve/pods/seaweedfs-64568bcd49-qvr5s/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046500006Z namespaces/kserve/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046532977Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046587348Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046630379Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046720132Z namespaces/kuadrant-system/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046770694Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046843196Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.046906298Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.04697698Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.047019041Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.047107993Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.047409072Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.04768305Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.047733742Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.047786703Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.047844955Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.047912256Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.047997559Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.04804273Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048087362Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048170084Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048216375Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048243496Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048298218Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.04839359Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048473833Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048561265Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048611087Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048660888Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048827903Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.048966407Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.049379829Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.049457621Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.049777041Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.049919995Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050148651Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050264155Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050304466Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050414159Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.05045371Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050516442Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050548293Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050638746Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050716918Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050761309Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050820571Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050875012Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050919243Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.050990036Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.051004936Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.051049607Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.051234673Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.051466929Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.051614864Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.05182325Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052059096Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052104918Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-w6pzh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052409367Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052457538Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052551491Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052634973Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052719816Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052781517Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052831269Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052903011Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.052974113Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.053079976Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.05322568Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.053322143Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.053402195Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.053415196Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.053452867Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino-7cf4765b85-gvscv.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.053539389Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.053548759Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.05355678Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.053608601Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.0642687Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064328771Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-gvscv/authorino/authorino/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064347062Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064404704Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/authorino-operator-56fb6f6856-b9spq.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064501216Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064513017Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064523177Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064576438Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064760904Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064826256Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-b9spq/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064903228Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.064951319Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/dns-operator-controller-manager-ccbdf686-dccw7.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065055462Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065063732Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065070463Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065114324Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065227047Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065295409Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-dccw7/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065352711Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065402362Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin-598f86468b-vrnns.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065477904Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065489305Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065504405Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065541686Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065621059Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065698531Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-vrnns/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065717142Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.065772123Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/kuadrant-operator-controller-manager-66866f5fcc-tpm8g.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.06600678Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.06601997Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.066026491Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.066065861Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074453014Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074516816Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-tpm8g/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074560167Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074601148Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador-limitador-85d9db4d5d-rn5v2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074691901Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074700331Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074704461Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074737712Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074830735Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074925328Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-rn5v2/limitador/limitador/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.074980679Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.07501241Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/limitador-operator-controller-manager-774d94476f-48g2h.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075122734Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075133634Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075163675Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075187525Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075325589Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075394951Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-48g2h/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075448413Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075481264Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075536775Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075583027Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075642449Z namespaces/kube-node-lease/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075650409Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075654549Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.07569849Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-128-242.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075791913Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-193.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075874515Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-135-98.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075955887Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.075969638Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.07602446Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.076314228Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.076524414Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.076667088Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.076888984Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.077126661Z namespaces/kube-public/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.077139322Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.077149482Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.077188923Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.077357548Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.077537293Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.077688728Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.077912934Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078147991Z namespaces/kube-system/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078195842Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078325726Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078372557Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078441309Z namespaces/kube-system/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.0784852Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078613834Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078690457Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078773069Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078838201Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078909123Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.078974895Z namespaces/kube-system/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079022376Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079113499Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.07915593Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079200961Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079282494Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079323225Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079333325Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079381917Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-yzwwrnnwmhu6xtzakjylqcqide.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079470809Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079554991Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079659214Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079791328Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.07985712Z namespaces/kube-system/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.079927012Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.080105587Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08018735Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.080451677Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.0805271Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.080819448Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.080972483Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081285381Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081340993Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081389644Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081447236Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081510658Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08157432Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081627091Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081706484Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081799636Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081853668Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08193236Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.081993602Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.082035713Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.082113776Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.082124276Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.082153137Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.082314851Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.082491116Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.082638481Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.082830276Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083064103Z namespaces/kube-system/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083078123Z namespaces/kube-system/pods/global-pull-secret-syncer-7pv5p/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083120545Z namespaces/kube-system/pods/global-pull-secret-syncer-7pv5p/global-pull-secret-syncer-7pv5p.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083199577Z namespaces/kube-system/pods/global-pull-secret-syncer-7pv5p/global-pull-secret-syncer/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083210447Z namespaces/kube-system/pods/global-pull-secret-syncer-7pv5p/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083218097Z namespaces/kube-system/pods/global-pull-secret-syncer-7pv5p/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083268059Z namespaces/kube-system/pods/global-pull-secret-syncer-7pv5p/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083588478Z namespaces/kube-system/pods/global-pull-secret-syncer-7pv5p/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08365049Z namespaces/kube-system/pods/global-pull-secret-syncer-7pv5p/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083717572Z namespaces/kube-system/pods/global-pull-secret-syncer-w9w6m/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083760803Z namespaces/kube-system/pods/global-pull-secret-syncer-w9w6m/global-pull-secret-syncer-w9w6m.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083834195Z namespaces/kube-system/pods/global-pull-secret-syncer-w9w6m/global-pull-secret-syncer/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083844995Z namespaces/kube-system/pods/global-pull-secret-syncer-w9w6m/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083851286Z namespaces/kube-system/pods/global-pull-secret-syncer-w9w6m/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.083874116Z namespaces/kube-system/pods/global-pull-secret-syncer-w9w6m/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084203516Z namespaces/kube-system/pods/global-pull-secret-syncer-w9w6m/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084273938Z namespaces/kube-system/pods/global-pull-secret-syncer-w9w6m/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084294299Z namespaces/kube-system/pods/global-pull-secret-syncer-x8j5c/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08435044Z namespaces/kube-system/pods/global-pull-secret-syncer-x8j5c/global-pull-secret-syncer-x8j5c.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084422322Z namespaces/kube-system/pods/global-pull-secret-syncer-x8j5c/global-pull-secret-syncer/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084432083Z namespaces/kube-system/pods/global-pull-secret-syncer-x8j5c/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084438713Z namespaces/kube-system/pods/global-pull-secret-syncer-x8j5c/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084476654Z namespaces/kube-system/pods/global-pull-secret-syncer-x8j5c/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084796703Z namespaces/kube-system/pods/global-pull-secret-syncer-x8j5c/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084879865Z namespaces/kube-system/pods/global-pull-secret-syncer-x8j5c/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084903386Z namespaces/kube-system/pods/konnectivity-agent-6dn6d/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.084964258Z namespaces/kube-system/pods/konnectivity-agent-6dn6d/konnectivity-agent-6dn6d.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08504474Z namespaces/kube-system/pods/konnectivity-agent-6dn6d/konnectivity-agent/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08505511Z namespaces/kube-system/pods/konnectivity-agent-6dn6d/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.085066071Z namespaces/kube-system/pods/konnectivity-agent-6dn6d/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.085110062Z namespaces/kube-system/pods/konnectivity-agent-6dn6d/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.086034559Z namespaces/kube-system/pods/konnectivity-agent-6dn6d/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.086108751Z namespaces/kube-system/pods/konnectivity-agent-6dn6d/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.086165193Z namespaces/kube-system/pods/konnectivity-agent-c6vx2/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.086182843Z namespaces/kube-system/pods/konnectivity-agent-c6vx2/konnectivity-agent-c6vx2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.086288236Z namespaces/kube-system/pods/konnectivity-agent-c6vx2/konnectivity-agent/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.086296686Z namespaces/kube-system/pods/konnectivity-agent-c6vx2/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.086305137Z namespaces/kube-system/pods/konnectivity-agent-c6vx2/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.086347918Z namespaces/kube-system/pods/konnectivity-agent-c6vx2/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.087311436Z namespaces/kube-system/pods/konnectivity-agent-c6vx2/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.087391968Z namespaces/kube-system/pods/konnectivity-agent-c6vx2/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08744615Z namespaces/kube-system/pods/konnectivity-agent-r4lqd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.087503811Z namespaces/kube-system/pods/konnectivity-agent-r4lqd/konnectivity-agent-r4lqd.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.087573833Z namespaces/kube-system/pods/konnectivity-agent-r4lqd/konnectivity-agent/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.087591064Z namespaces/kube-system/pods/konnectivity-agent-r4lqd/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.087597184Z namespaces/kube-system/pods/konnectivity-agent-r4lqd/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.087628425Z namespaces/kube-system/pods/konnectivity-agent-r4lqd/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.088442908Z namespaces/kube-system/pods/konnectivity-agent-r4lqd/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.088514041Z namespaces/kube-system/pods/konnectivity-agent-r4lqd/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.088545531Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.088596473Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.088681626Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal/haproxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.088691136Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.088701886Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.088755308Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089060306Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089129588Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-242.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08918705Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089215631Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089309333Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal/haproxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089317884Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089321924Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089361295Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089625183Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089695995Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089742886Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089791578Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08987815Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal/haproxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.08989675Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089903351Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.089943832Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090175179Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090244011Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-98.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090276612Z namespaces/kube-system/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090315963Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090373684Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090383795Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090423776Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090521109Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.09056937Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090623362Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090671443Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090700304Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090755455Z namespaces/observability/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090764056Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090778776Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.090825477Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.091002642Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.091179028Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.091317052Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.091520917Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.091753444Z namespaces/open-cluster-management-35813af0-8a90-4d19-abf5-d2d9edb86/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.091762304Z namespaces/open-cluster-management-35813af0-8a90-4d19-abf5-d2d9edb86/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.091777115Z namespaces/open-cluster-management-35813af0-8a90-4d19-abf5-d2d9edb86/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.091836867Z namespaces/open-cluster-management-35813af0-8a90-4d19-abf5-d2d9edb86/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.092024812Z namespaces/open-cluster-management-35813af0-8a90-4d19-abf5-d2d9edb86/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.092245969Z namespaces/open-cluster-management-35813af0-8a90-4d19-abf5-d2d9edb86/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.092387223Z namespaces/open-cluster-management-35813af0-8a90-4d19-abf5-d2d9edb86/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.092581808Z namespaces/open-cluster-management-35813af0-8a90-4d19-abf5-d2d9edb86/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.092798274Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.092811835Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.092819215Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.092829565Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.09298613Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.093021811Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.093027511Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.093072042Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.093241577Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.093430403Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.093566487Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.093772673Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.09401923Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094070671Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094145264Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094177744Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094256767Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094303448Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094390331Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094474783Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094557285Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094733561Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094762801Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094846274Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094889295Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.094961497Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.095004878Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.095036709Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.095133002Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.095184934Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.095225065Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.096169612Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.096250634Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.096334757Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.09643141Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.096506992Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.096633415Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.096831771Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.096916684Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.096969655Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097014696Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097064698Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.09712667Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097170661Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097257323Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097341266Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097393287Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097440559Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097504901Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097554662Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097606024Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097653225Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097711196Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097718457Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097763188Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.097993235Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.098330785Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.098468749Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.098674444Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.09885653Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.098912121Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.098964863Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099001524Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099102577Z namespaces/openshift-apiserver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099148278Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.09922177Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099264721Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099311713Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099347394Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099434586Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099513869Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099591701Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099645032Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099684024Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099725735Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099782336Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099883919Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099944901Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.099998463Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100076565Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100153097Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100200728Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100304842Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100385464Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100468776Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100549999Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100632261Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100772705Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10095823Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.100998642Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101051173Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101112945Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101167196Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101228118Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10127878Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101357652Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101439014Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101498026Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101547267Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101600699Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10165215Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101681431Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.101882777Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10197802Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.102127344Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.102303789Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.102443133Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.102639449Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.102816094Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.102851655Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.102939228Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.102983209Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103037961Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103046491Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103053131Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103097272Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103172394Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103182235Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103229236Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103380711Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103565256Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10371963Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.103937767Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.104125632Z namespaces/openshift-authentication/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.104133672Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.104140373Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.104193554Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.104347979Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.104526054Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.104672468Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.104882074Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10507949Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10509014Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10509621Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.105133521Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.105209664Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.105218544Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.105270785Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.105420099Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.105599845Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.105744189Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.105957665Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106156891Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106166151Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106171031Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106198202Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106265764Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106275504Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106324316Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106383077Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106396048Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10645746Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106615504Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.106877452Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107020176Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107231832Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107429178Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107478899Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107543261Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107580812Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107658714Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107697475Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107784638Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107880561Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.107972503Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108018965Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108070956Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108110157Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108164019Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108245081Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108292993Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108340284Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108422646Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108470248Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108480198Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108534689Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108614042Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108662933Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108784787Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.10887853Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.108975632Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109050555Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109136337Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109286801Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109421345Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109469597Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109517288Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109569519Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109618311Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109661882Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109710714Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109789646Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.109825787Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110004322Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110079384Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110131096Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110171167Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110205668Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110261379Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110305531Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110313231Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110352442Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110501997Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110678831Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.110820276Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111040462Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111233368Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111275529Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11132219Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1113297Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111372292Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111421913Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111469214Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111523866Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111566127Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1116515Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111696591Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111819235Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111875456Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.111959389Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1120028Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112132214Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112211886Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112292558Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11233968Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112383621Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112438213Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112486864Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112564456Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112621178Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112668489Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112756982Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112814603Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112822584Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112887626Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.112978888Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.113074391Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.113177704Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.113269206Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.113316778Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.113413491Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.113497863Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.113853163Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.113958557Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114167683Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114321037Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114431Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114704008Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114732709Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114800511Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114874323Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114931794Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.114983116Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115021437Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115069789Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115150341Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115229873Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115276804Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115316386Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115367307Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115415228Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115484581Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115501421Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115549112Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115699357Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.115881752Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116035697Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116232902Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116428678Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116436748Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11649616Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/aws-ebs-csi-driver-node-47wpr.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116595623Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-driver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116606363Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-driver/csi-driver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116613053Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-driver/csi-driver/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116649154Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116762938Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11683441Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116879801Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-liveness-probe/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116892301Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116901802Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.116955993Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117066086Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117113418Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117172209Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-node-driver-registrar/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11718218Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11718729Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117215201Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117336954Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117400976Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-47wpr/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117454978Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117474338Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/aws-ebs-csi-driver-node-8rn5f.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117598232Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-driver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117613542Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-driver/csi-driver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117621012Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-driver/csi-driver/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117639343Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117755186Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117823398Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117844369Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-liveness-probe/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117852929Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11789234Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.117988403Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118073105Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118140137Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118181999Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-node-driver-registrar/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118189639Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118193799Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11823375Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118330733Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118400195Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8rn5f/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118467967Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118510698Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/aws-ebs-csi-driver-node-rhqrq.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118601491Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-driver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118608811Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-driver/csi-driver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118619461Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-driver/csi-driver/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118659622Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118750935Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118819137Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118888709Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-liveness-probe/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118901209Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.118963081Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119013123Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119104715Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119172587Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119215018Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-node-driver-registrar/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119228599Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119236079Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11928422Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119380413Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119448595Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rhqrq/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119490386Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119521907Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119582869Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119590819Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119640511Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119702452Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119737524Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119811616Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119853427Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.119939129Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11995762Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.11996435Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120000061Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120058383Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120066133Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120117644Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120165626Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120200337Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120243328Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120396053Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120793334Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.120951279Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.121151065Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.121362391Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.121412582Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.121542886Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.121572737Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.121652009Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12169608Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.121800023Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.121948128Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1220339Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122080611Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122128033Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122208125Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122248836Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122316468Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122344549Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122440552Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122493873Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122523984Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122627407Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12272127Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12272949Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122777932Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122847503Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.122897475Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.123794661Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.123915474Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12409551Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.124176642Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.124389008Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.124542692Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.124681487Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.124766339Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.124823131Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.124888002Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.124949394Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125015286Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125044397Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125128209Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12516245Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125261623Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125339875Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125401607Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125458379Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12550688Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125560762Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125614063Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125648144Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125718617Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125767638Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125813269Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12582602Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.125968704Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.126161929Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.126356385Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.126516389Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.126738936Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.126964872Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.126977103Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-65x8z/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127019054Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-65x8z/tuned-65x8z.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127126007Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-65x8z/tuned/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127135687Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-65x8z/tuned/tuned/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127139858Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-65x8z/tuned/tuned/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127190409Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-65x8z/tuned/tuned/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127307102Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-65x8z/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127377924Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-65x8z/tuned/tuned/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127429506Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-86ljg/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127457117Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-86ljg/tuned-86ljg.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12756094Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-86ljg/tuned/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12756807Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-86ljg/tuned/tuned/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12757439Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-86ljg/tuned/tuned/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127624492Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-86ljg/tuned/tuned/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127745185Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-86ljg/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127817157Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-86ljg/tuned/tuned/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12790953Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gcmzb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.127967782Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gcmzb/tuned-gcmzb.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128081755Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gcmzb/tuned/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128089065Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gcmzb/tuned/tuned/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128093195Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gcmzb/tuned/tuned/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128132446Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gcmzb/tuned/tuned/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128250159Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gcmzb/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128317801Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gcmzb/tuned/tuned/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128363763Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128404324Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128445715Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128494797Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128528958Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128537998Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128592789Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-128-242.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128688712Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-193.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128798855Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-135-98.ec2.internal.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128831016Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.128914279Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129015412Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129065223Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129131805Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129178377Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129222268Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12929621Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129384432Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129503736Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129605109Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.12963429Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129688901Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129750383Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129779624Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129880347Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.129959309Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13000587Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.130097003Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.130166985Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.130216607Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.130397882Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.130480664Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.130616028Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13069604Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.130817564Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.130966218Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131157674Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131222336Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131273237Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131331709Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13138029Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131443922Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131492973Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131570466Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131650368Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.131688329Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13174004Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13206936Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.132114081Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.132201344Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.132245595Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.132309127Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.132351788Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13242468Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.132431751Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.132469052Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13275676Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.132967046Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13310628Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133299926Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133481101Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133490561Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133526152Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator-6748489656-x97fb.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133613295Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator-watch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133620725Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133624865Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133663286Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133759769Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133828061Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133874682Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.133884833Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.134007336Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.134053067Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.134674485Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.134744927Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-x97fb/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.134784868Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.134818989Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.134874321Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.134938313Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135036846Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135080657Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135153409Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13519256Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135268602Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135313354Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135400026Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135501679Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135602572Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135642593Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135672964Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135748236Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135782817Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13587706Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135941522Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.135981613Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136062056Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136118917Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136128157Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136184859Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136272291Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136344134Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136393005Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136515929Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136600961Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13690852Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.136991582Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137100026Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1372591Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137397684Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137481006Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137512598Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137583269Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13761313Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137689903Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137725493Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137797636Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.137827276Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13794846Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138024502Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138086114Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138139005Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138197697Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138253699Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138337631Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138391293Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138448294Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138500386Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138569838Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138604569Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138654231Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.138803085Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139000721Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139137885Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13933955Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139518156Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139527346Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-z9cvg/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139578697Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-z9cvg/volume-data-source-validator-549c6d54cf-z9cvg.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13966589Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-z9cvg/volume-data-source-validator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13967573Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-z9cvg/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.13968977Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-z9cvg/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139738342Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-z9cvg/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139843135Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-z9cvg/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139930677Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-z9cvg/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.139959738Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14000435Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140051421Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140058401Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140087102Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140148014Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140317848Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140327029Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140330989Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14037437Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140419811Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140492104Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140533655Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140578346Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140640678Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14072045Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140816263Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140939277Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.140966087Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141026189Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14106184Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141096761Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141186674Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141221035Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141275736Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141354409Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14141298Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141421711Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141432361Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141540264Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141585975Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141688508Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141767781Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.141881614Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142032298Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14210951Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142239494Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142408409Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142468381Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142517822Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142565864Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142608545Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142664607Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142713318Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14279517Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142878043Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142950645Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.142984296Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.143052838Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.143094559Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14314066Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.143159151Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.143218032Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.143371057Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.143557242Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.143698406Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.143916703Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144213171Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144258833Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144322544Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144364886Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144460138Z namespaces/openshift-config-managed/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14450665Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144568722Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144609283Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144681215Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144709315Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144817759Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144909891Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.144994204Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145024645Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145073246Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145132698Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145180629Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145265852Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145320553Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145371295Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145450547Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.145523609Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1455655Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.148764723Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.148879796Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14899683Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149077572Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149185485Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149324689Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149433242Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149484434Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149543845Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149631338Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.14969315Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149728621Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149801333Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.149841384Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150010309Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15005054Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150171933Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150256776Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150313268Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150364929Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150421081Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150454572Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150525304Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150532714Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150609546Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.150792911Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.151026218Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.151184723Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.151389089Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.151663047Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.151704248Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15177007Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15177925Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.151830541Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.151908804Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.151947415Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.152011597Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.152055768Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15211657Z namespaces/openshift-config-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15212575Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1521322Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.152170181Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.152221663Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.152233633Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.152295175Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15245609Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.152636335Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.152790959Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153020046Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153404527Z namespaces/openshift-config/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153454679Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153536011Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153583622Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153650944Z namespaces/openshift-config/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153682595Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153783258Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153885131Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.153976903Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154020515Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154055426Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154112558Z namespaces/openshift-config/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154148588Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154227981Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154261562Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154311263Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154391235Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154471948Z namespaces/openshift-config/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154501009Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154633682Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154708325Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154796487Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15488476Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.154974643Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155116076Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155255531Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155302632Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155359114Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155389944Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155439406Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155474767Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155523058Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155554709Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155608631Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155691113Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155776776Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155811747Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155876478Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15592871Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.155975801Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.156006902Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.156021783Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.156083855Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.156238369Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.156415814Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.156551858Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.156770444Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15697408Z namespaces/openshift-config/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157009631Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157077803Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157109974Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157211447Z namespaces/openshift-console-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157248508Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.15732562Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157361321Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157437304Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157471455Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157565577Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157686611Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157789304Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157822435Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157856806Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157938418Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.157977769Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158059742Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158121043Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158169235Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158245907Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158371051Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158382021Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158408062Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158473564Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.158502745Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.159355819Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.159442612Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.159680048Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.159759501Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.159892115Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160031269Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160233524Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160288936Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160321797Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160386139Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16043162Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160474811Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160515733Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160596935Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160678587Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160786651Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160816211Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160889014Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.160946505Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161008197Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161047888Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1611146Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161155451Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161217093Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161224133Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161275425Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161426749Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161603434Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161741298Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.161976465Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.162244653Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.162251753Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-27xfm/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.162291454Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-27xfm/console-operator-584dcb6b54-27xfm.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.162385097Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-27xfm/console-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.162394837Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-27xfm/console-operator/console-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.162399857Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-27xfm/console-operator/console-operator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.162441198Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-27xfm/console-operator/console-operator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.162950253Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-27xfm/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.163008165Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.163054656Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.163136988Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.163165519Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16318434Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16319594Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16320653Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.163278883Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.163440717Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.163620882Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.163760977Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.164001164Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16422298Z namespaces/openshift-console/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.164255941Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16457405Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.164617401Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.164674493Z namespaces/openshift-console/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.164714514Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.164813877Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165005282Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16526392Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165307121Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165341722Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165393164Z namespaces/openshift-console/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165423415Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165517607Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165571019Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16562144Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165700693Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165776115Z namespaces/openshift-console/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.165822226Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16664667Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.166735492Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167042452Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167124764Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16733879Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167478384Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167659799Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167716371Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167770442Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167840224Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167913407Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.167949118Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168015009Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168105582Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168182824Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168229306Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168277777Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168334869Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16837411Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168443812Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168462673Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168544175Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168561615Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168606487Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.168776462Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169005858Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169185303Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169398349Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169595555Z namespaces/openshift-console/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169603535Z namespaces/openshift-console/pods/console-54bbdcdd48-chw9c/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169657847Z namespaces/openshift-console/pods/console-54bbdcdd48-chw9c/console-54bbdcdd48-chw9c.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16976267Z namespaces/openshift-console/pods/console-54bbdcdd48-chw9c/console/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.16977169Z namespaces/openshift-console/pods/console-54bbdcdd48-chw9c/console/console/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169776631Z namespaces/openshift-console/pods/console-54bbdcdd48-chw9c/console/console/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169806681Z namespaces/openshift-console/pods/console-54bbdcdd48-chw9c/console/console/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.169967666Z namespaces/openshift-console/pods/console-54bbdcdd48-chw9c/console/console/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170044048Z namespaces/openshift-console/pods/console-54bbdcdd48-chw9c/console/console/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17009695Z namespaces/openshift-console/pods/downloads-7c97dc4555-4qmvc/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170132931Z namespaces/openshift-console/pods/downloads-7c97dc4555-4qmvc/downloads-7c97dc4555-4qmvc.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170259674Z namespaces/openshift-console/pods/downloads-7c97dc4555-4qmvc/download-server/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170271605Z namespaces/openshift-console/pods/downloads-7c97dc4555-4qmvc/download-server/download-server/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170277755Z namespaces/openshift-console/pods/downloads-7c97dc4555-4qmvc/download-server/download-server/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170287785Z namespaces/openshift-console/pods/downloads-7c97dc4555-4qmvc/download-server/download-server/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170663276Z namespaces/openshift-console/pods/downloads-7c97dc4555-4qmvc/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170729498Z namespaces/openshift-console/pods/downloads-7c97dc4555-4qmvc/download-server/download-server/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170746608Z namespaces/openshift-console/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17080607Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170858232Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.170931794Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171018456Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171062168Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1711421Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171163131Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171251783Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171270964Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171372667Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171456049Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171539342Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171581893Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171637184Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171692756Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171742117Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1718233Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171877621Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.171935043Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172022196Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172083967Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172130459Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172259452Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172337705Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172422787Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172507779Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172582342Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172708475Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172918551Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.172971893Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173004894Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173068206Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173112727Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173184599Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17322556Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173315353Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173406516Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173437356Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173481688Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173537659Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17357294Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173624202Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173649123Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173700034Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.173854058Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.174189648Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.174314422Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.174520448Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.174702033Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.174740234Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.174804486Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.174827626Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17493689Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.174972411Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175063123Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175101145Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175170427Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175199417Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17529569Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175381482Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175463285Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175544657Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175592939Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17563436Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175673391Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175765484Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175799895Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175829595Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.175949109Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1759979Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.176196326Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.176238357Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17633521Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17635161Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.176478314Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.176562307Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.176637619Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.176724211Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.176814794Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.176970208Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177131383Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177191895Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177231146Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177291618Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177337799Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17737353Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177433802Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177509174Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177590106Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177633027Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177677649Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177733441Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177787282Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177822133Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177834993Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.177922576Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.178212104Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17839324Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.178530304Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.178734999Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.178937385Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.178968816Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179025778Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179071169Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179145761Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179192523Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179253594Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179294026Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179371328Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179413599Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179500212Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179578014Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179657476Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179705098Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179738699Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.17979369Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179820531Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179943694Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.179963075Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180015767Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180105929Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180177591Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180212622Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180318925Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180405898Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18049375Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180579533Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180654075Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180791729Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.180989805Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181058627Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181109078Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18116739Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181215491Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181244892Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181300373Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181382106Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181462188Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18152201Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181571312Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181637963Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181689435Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181744786Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181793408Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.181824059Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18187835Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.182053625Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.182064416Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.182087156Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.182244101Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.182431786Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18256733Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.182798177Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.183001083Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.183034204Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.183089265Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.183120086Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.183218969Z namespaces/openshift-dns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18325606Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.183362243Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.183397644Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184176997Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184209908Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184369202Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184532477Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184609779Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1846409Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184695552Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184735093Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184791605Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184883007Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.184948159Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18497775Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.185076493Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.185151995Z namespaces/openshift-dns/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.185183546Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.185304259Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.185387712Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.185610298Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.185690151Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.185979249Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186098092Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186277597Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18634319Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18637784Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186442902Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186477543Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186532365Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186561006Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186651838Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186735811Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186803803Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186844014Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186919506Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.186969528Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18703439Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.187069871Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.187134072Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.187171323Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.187239005Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18739068Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.187568375Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.187703359Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.187958206Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188167932Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188178323Z namespaces/openshift-dns/pods/dns-default-25gll/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188212204Z namespaces/openshift-dns/pods/dns-default-25gll/dns-default-25gll.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188300356Z namespaces/openshift-dns/pods/dns-default-25gll/dns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188308366Z namespaces/openshift-dns/pods/dns-default-25gll/dns/dns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188312516Z namespaces/openshift-dns/pods/dns-default-25gll/dns/dns/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188337807Z namespaces/openshift-dns/pods/dns-default-25gll/dns/dns/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18844552Z namespaces/openshift-dns/pods/dns-default-25gll/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188510262Z namespaces/openshift-dns/pods/dns-default-25gll/dns/dns/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188554453Z namespaces/openshift-dns/pods/dns-default-25gll/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188564174Z namespaces/openshift-dns/pods/dns-default-25gll/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188574674Z namespaces/openshift-dns/pods/dns-default-25gll/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188603855Z namespaces/openshift-dns/pods/dns-default-25gll/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188708758Z namespaces/openshift-dns/pods/dns-default-25gll/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.18879295Z namespaces/openshift-dns/pods/dns-default-25gll/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188845732Z namespaces/openshift-dns/pods/dns-default-wt4mt/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.188914354Z namespaces/openshift-dns/pods/dns-default-wt4mt/dns-default-wt4mt.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189004096Z namespaces/openshift-dns/pods/dns-default-wt4mt/dns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189011316Z namespaces/openshift-dns/pods/dns-default-wt4mt/dns/dns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189021717Z namespaces/openshift-dns/pods/dns-default-wt4mt/dns/dns/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189069068Z namespaces/openshift-dns/pods/dns-default-wt4mt/dns/dns/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189175231Z namespaces/openshift-dns/pods/dns-default-wt4mt/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189242873Z namespaces/openshift-dns/pods/dns-default-wt4mt/dns/dns/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189275644Z namespaces/openshift-dns/pods/dns-default-wt4mt/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189282984Z namespaces/openshift-dns/pods/dns-default-wt4mt/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189286904Z namespaces/openshift-dns/pods/dns-default-wt4mt/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189336756Z namespaces/openshift-dns/pods/dns-default-wt4mt/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189432419Z namespaces/openshift-dns/pods/dns-default-wt4mt/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189503121Z namespaces/openshift-dns/pods/dns-default-wt4mt/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189554882Z namespaces/openshift-dns/pods/dns-default-xxsch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189583903Z namespaces/openshift-dns/pods/dns-default-xxsch/dns-default-xxsch.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189690396Z namespaces/openshift-dns/pods/dns-default-xxsch/dns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189698727Z namespaces/openshift-dns/pods/dns-default-xxsch/dns/dns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189702676Z namespaces/openshift-dns/pods/dns-default-xxsch/dns/dns/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189750948Z namespaces/openshift-dns/pods/dns-default-xxsch/dns/dns/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.189849101Z namespaces/openshift-dns/pods/dns-default-xxsch/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190058067Z namespaces/openshift-dns/pods/dns-default-xxsch/dns/dns/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190118909Z namespaces/openshift-dns/pods/dns-default-xxsch/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190127289Z namespaces/openshift-dns/pods/dns-default-xxsch/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190133629Z namespaces/openshift-dns/pods/dns-default-xxsch/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.19018225Z namespaces/openshift-dns/pods/dns-default-xxsch/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190277813Z namespaces/openshift-dns/pods/dns-default-xxsch/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190352975Z namespaces/openshift-dns/pods/dns-default-xxsch/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190402487Z namespaces/openshift-dns/pods/node-resolver-mbdtd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190452808Z namespaces/openshift-dns/pods/node-resolver-mbdtd/node-resolver-mbdtd.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190547591Z namespaces/openshift-dns/pods/node-resolver-mbdtd/dns-node-resolver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190558101Z namespaces/openshift-dns/pods/node-resolver-mbdtd/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190563861Z namespaces/openshift-dns/pods/node-resolver-mbdtd/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190591752Z namespaces/openshift-dns/pods/node-resolver-mbdtd/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190696145Z namespaces/openshift-dns/pods/node-resolver-mbdtd/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190770727Z namespaces/openshift-dns/pods/node-resolver-mbdtd/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190820549Z namespaces/openshift-dns/pods/node-resolver-ts59j/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190887711Z namespaces/openshift-dns/pods/node-resolver-ts59j/node-resolver-ts59j.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190972763Z namespaces/openshift-dns/pods/node-resolver-ts59j/dns-node-resolver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190981884Z namespaces/openshift-dns/pods/node-resolver-ts59j/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.190992334Z namespaces/openshift-dns/pods/node-resolver-ts59j/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191043365Z namespaces/openshift-dns/pods/node-resolver-ts59j/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191138888Z namespaces/openshift-dns/pods/node-resolver-ts59j/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.19121703Z namespaces/openshift-dns/pods/node-resolver-ts59j/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191254301Z namespaces/openshift-dns/pods/node-resolver-whv2s/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191305073Z namespaces/openshift-dns/pods/node-resolver-whv2s/node-resolver-whv2s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191398716Z namespaces/openshift-dns/pods/node-resolver-whv2s/dns-node-resolver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191407816Z namespaces/openshift-dns/pods/node-resolver-whv2s/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191418446Z namespaces/openshift-dns/pods/node-resolver-whv2s/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191459697Z namespaces/openshift-dns/pods/node-resolver-whv2s/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1915487Z namespaces/openshift-dns/pods/node-resolver-whv2s/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191632032Z namespaces/openshift-dns/pods/node-resolver-whv2s/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191646773Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191697634Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191743015Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191794757Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191856989Z namespaces/openshift-etcd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191929171Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.191997643Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192022724Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192098366Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192156987Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1922324Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192320232Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192396685Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192427315Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192483847Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192544469Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1925919Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192679643Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192761525Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192786206Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192879208Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.192965751Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193010712Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193117445Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193196028Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.19327638Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193358482Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193439605Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193580999Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193770664Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193832796Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.193896138Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.19397444Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194022081Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194089114Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194115944Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194216857Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194293829Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194338331Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194368062Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194441614Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194484425Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194536026Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194543387Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194583128Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194743622Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.194962569Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195102463Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195300149Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195496934Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195540925Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195592717Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195635788Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195729011Z namespaces/openshift-host-network/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195774152Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195849834Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195911896Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.195997919Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.19604672Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196129673Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196209735Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196289977Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196346319Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.19638267Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196459202Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196496063Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196587066Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196628487Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196673298Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196768941Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196854233Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.196927856Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197049849Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197127741Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197208384Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197289666Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197371688Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197558444Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197803671Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197873443Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.197929405Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198003857Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198031808Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.19811168Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.19813035Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198205342Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198228993Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198330616Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198405828Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198429859Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198486631Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198545932Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198599504Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198619965Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198666266Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198716687Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.198900513Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.199080848Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.199226692Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.199456979Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.199640384Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.199654864Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.199718486Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.199761838Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.1998526Z namespaces/openshift-image-registry/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.199920222Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200004065Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200043356Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200097567Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200130468Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200250072Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200380815Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200581111Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200613172Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200662433Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200694085Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200760546Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20088915Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200961862Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.200984863Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.201093076Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.201112726Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.201119537Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.201173678Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20124238Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.201291062Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.202203508Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.202332502Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20261099Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.202691612Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.2029591Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203106564Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20329801Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203354611Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203406673Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203474955Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203529456Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203587128Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20364133Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203717002Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203804724Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203877057Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.203935368Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20399926Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204033461Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204135424Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204220576Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204262328Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204320349Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204368261Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204413232Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204463074Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204476834Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204527965Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20469258Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.204890956Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205103182Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205336619Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205545575Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205552985Z namespaces/openshift-image-registry/pods/image-pruner-29756160-sjnbv/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205601986Z namespaces/openshift-image-registry/pods/image-pruner-29756160-sjnbv/image-pruner-29756160-sjnbv.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205698749Z namespaces/openshift-image-registry/pods/image-pruner-29756160-sjnbv/image-pruner/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205706269Z namespaces/openshift-image-registry/pods/image-pruner-29756160-sjnbv/image-pruner/image-pruner/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205710419Z namespaces/openshift-image-registry/pods/image-pruner-29756160-sjnbv/image-pruner/image-pruner/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20574478Z namespaces/openshift-image-registry/pods/image-pruner-29756160-sjnbv/image-pruner/image-pruner/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205844403Z namespaces/openshift-image-registry/pods/image-pruner-29756160-sjnbv/image-pruner/image-pruner/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205934326Z namespaces/openshift-image-registry/pods/image-pruner-29756160-sjnbv/image-pruner/image-pruner/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.205973307Z namespaces/openshift-image-registry/pods/image-registry-7778c94ff4-8ns4j/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.206010258Z namespaces/openshift-image-registry/pods/image-registry-7778c94ff4-8ns4j/image-registry-7778c94ff4-8ns4j.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.206118261Z namespaces/openshift-image-registry/pods/image-registry-7778c94ff4-8ns4j/registry/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.206128172Z namespaces/openshift-image-registry/pods/image-registry-7778c94ff4-8ns4j/registry/registry/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.206138502Z namespaces/openshift-image-registry/pods/image-registry-7778c94ff4-8ns4j/registry/registry/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.206179823Z namespaces/openshift-image-registry/pods/image-registry-7778c94ff4-8ns4j/registry/registry/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208187881Z namespaces/openshift-image-registry/pods/image-registry-7778c94ff4-8ns4j/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208250413Z namespaces/openshift-image-registry/pods/image-registry-7778c94ff4-8ns4j/registry/registry/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208268844Z namespaces/openshift-image-registry/pods/node-ca-8tftg/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208324465Z namespaces/openshift-image-registry/pods/node-ca-8tftg/node-ca-8tftg.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208417748Z namespaces/openshift-image-registry/pods/node-ca-8tftg/node-ca/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208431668Z namespaces/openshift-image-registry/pods/node-ca-8tftg/node-ca/node-ca/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208439509Z namespaces/openshift-image-registry/pods/node-ca-8tftg/node-ca/node-ca/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208454159Z namespaces/openshift-image-registry/pods/node-ca-8tftg/node-ca/node-ca/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208613354Z namespaces/openshift-image-registry/pods/node-ca-8tftg/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208667475Z namespaces/openshift-image-registry/pods/node-ca-8tftg/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208693746Z namespaces/openshift-image-registry/pods/node-ca-hg8hl/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208731137Z namespaces/openshift-image-registry/pods/node-ca-hg8hl/node-ca-hg8hl.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20883231Z namespaces/openshift-image-registry/pods/node-ca-hg8hl/node-ca/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20884303Z namespaces/openshift-image-registry/pods/node-ca-hg8hl/node-ca/node-ca/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208856241Z namespaces/openshift-image-registry/pods/node-ca-hg8hl/node-ca/node-ca/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.208906132Z namespaces/openshift-image-registry/pods/node-ca-hg8hl/node-ca/node-ca/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209092357Z namespaces/openshift-image-registry/pods/node-ca-hg8hl/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209152069Z namespaces/openshift-image-registry/pods/node-ca-hg8hl/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20916943Z namespaces/openshift-image-registry/pods/node-ca-r8cns/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209219361Z namespaces/openshift-image-registry/pods/node-ca-r8cns/node-ca-r8cns.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209294493Z namespaces/openshift-image-registry/pods/node-ca-r8cns/node-ca/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209302773Z namespaces/openshift-image-registry/pods/node-ca-r8cns/node-ca/node-ca/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209312234Z namespaces/openshift-image-registry/pods/node-ca-r8cns/node-ca/node-ca/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209374076Z namespaces/openshift-image-registry/pods/node-ca-r8cns/node-ca/node-ca/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209546481Z namespaces/openshift-image-registry/pods/node-ca-r8cns/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209612402Z namespaces/openshift-image-registry/pods/node-ca-r8cns/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209643723Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209676974Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209744606Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209772387Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209831979Z namespaces/openshift-infra/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209838999Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.209842969Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.20989556Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.210055335Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21023204Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.210369394Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.2105728Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.210787496Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.210817387Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21091835Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.210957501Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211036144Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211076965Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211184398Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21126163Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211341242Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211397034Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211430625Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211502117Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211544958Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211630471Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211667852Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211716353Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211796906Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.211891328Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21195062Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.212050993Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.212145926Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21229621Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.212372512Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.212539727Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21265672Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.212881887Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.212919538Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21298383Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.213032041Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.213059612Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21368954Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.213726111Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.213822144Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.213927977Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.213961438Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.213995639Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.214056331Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.214084232Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.214284098Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.214298068Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.214344259Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.214508724Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21470091Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.214836863Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215179863Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215385469Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215396Z namespaces/openshift-ingress-canary/pods/ingress-canary-4hpqx/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215435241Z namespaces/openshift-ingress-canary/pods/ingress-canary-4hpqx/ingress-canary-4hpqx.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215517143Z namespaces/openshift-ingress-canary/pods/ingress-canary-4hpqx/serve-healthcheck-canary/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215526563Z namespaces/openshift-ingress-canary/pods/ingress-canary-4hpqx/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215531474Z namespaces/openshift-ingress-canary/pods/ingress-canary-4hpqx/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215578135Z namespaces/openshift-ingress-canary/pods/ingress-canary-4hpqx/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215681338Z namespaces/openshift-ingress-canary/pods/ingress-canary-4hpqx/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21575196Z namespaces/openshift-ingress-canary/pods/ingress-canary-4hpqx/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215810572Z namespaces/openshift-ingress-canary/pods/ingress-canary-hlgwc/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215830002Z namespaces/openshift-ingress-canary/pods/ingress-canary-hlgwc/ingress-canary-hlgwc.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215956396Z namespaces/openshift-ingress-canary/pods/ingress-canary-hlgwc/serve-healthcheck-canary/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215972626Z namespaces/openshift-ingress-canary/pods/ingress-canary-hlgwc/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.215982456Z namespaces/openshift-ingress-canary/pods/ingress-canary-hlgwc/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216027728Z namespaces/openshift-ingress-canary/pods/ingress-canary-hlgwc/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21611772Z namespaces/openshift-ingress-canary/pods/ingress-canary-hlgwc/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216192343Z namespaces/openshift-ingress-canary/pods/ingress-canary-hlgwc/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216219043Z namespaces/openshift-ingress-canary/pods/ingress-canary-wf4k2/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216278365Z namespaces/openshift-ingress-canary/pods/ingress-canary-wf4k2/ingress-canary-wf4k2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216355107Z namespaces/openshift-ingress-canary/pods/ingress-canary-wf4k2/serve-healthcheck-canary/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216365308Z namespaces/openshift-ingress-canary/pods/ingress-canary-wf4k2/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216388258Z namespaces/openshift-ingress-canary/pods/ingress-canary-wf4k2/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21643425Z namespaces/openshift-ingress-canary/pods/ingress-canary-wf4k2/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216531812Z namespaces/openshift-ingress-canary/pods/ingress-canary-wf4k2/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216598774Z namespaces/openshift-ingress-canary/pods/ingress-canary-wf4k2/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216629595Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216674256Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216736408Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.2167879Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216914563Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.216967965Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217042137Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217085839Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217153121Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217185821Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217279114Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217356006Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217439929Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21749674Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217539752Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217604294Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217654165Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217734037Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217773698Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.21782073Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.217931083Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.218002675Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.218048406Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.218933442Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219016214Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219124187Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219229861Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219311363Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219459717Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219681143Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219738625Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219790447Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219853359Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219934961Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.219993532Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220004023Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220055724Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220130787Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220151787Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.2202442Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220344163Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220396634Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220459726Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220502757Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220545739Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220623501Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220671022Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220771245Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220808936Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220884148Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220900009Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.220957371Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.221045213Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.221055643Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.221104635Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.221262209Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.221440225Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.221577248Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.221917038Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222141225Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222183896Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222247218Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222289589Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222391682Z namespaces/openshift-ingress/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222438113Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222510075Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222553737Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222618619Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.22266007Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.222746622Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.22299683Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223194435Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223248107Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223285388Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.22335322Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223388211Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223475953Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223532815Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223582526Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223664619Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.22372121Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223732251Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223785372Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223876845Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.223992818Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.224100641Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.224144373Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.224563025Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.224659978Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.224911855Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.224998787Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225280256Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225405059Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225608025Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225701628Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225733058Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225850672Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225883283Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225916334Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.225995806Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226034677Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226174431Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226207212Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226298155Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226377287Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226438369Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.22648376Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226542982Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226580273Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226651365Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226704137Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226741447Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226773348Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.22683959Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226853021Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.226916503Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.227087417Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.227267773Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.227404127Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.227609823Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.227813898Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.227821699Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.227889501Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/istiod-openshift-gateway-5677ccb49b-dz27l.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.228055076Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.228063006Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.228066976Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.228118117Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.236638544Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.236705946Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-dz27l/discovery/discovery/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.236774318Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.236817309Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.236977043Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.236988634Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.236992814Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.237039215Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238146467Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.23822648Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238275271Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router-default-6d7fb7b5bf-sg26j.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238390964Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238398955Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238411505Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238464897Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.23857177Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238638982Z namespaces/openshift-ingress/pods/router-default-6d7fb7b5bf-sg26j/router/router/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238691353Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238723334Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238792526Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238824327Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.238950001Z namespaces/openshift-insights/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239009162Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239091735Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239138836Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239210228Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239242579Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239372433Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239481636Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239586499Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239639891Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239668781Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239759384Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239792735Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239873797Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.239928979Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.240017031Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.240078593Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.240121345Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.240200887Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.240283269Z namespaces/openshift-insights/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24031802Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.241268288Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24134575Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.241631078Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.241796853Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242152423Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242235116Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24239682Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242462092Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242499383Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242569965Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242620927Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242707809Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242777361Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242826373Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242906825Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.242961447Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243020908Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243060699Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243154582Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243164982Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243255735Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243332877Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243390219Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24344091Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243505582Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243553894Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243623746Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243655147Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243739479Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24376988Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243822112Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243833512Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.243945325Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.244135721Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.244357917Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.244504381Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.244759019Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.245013086Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.245024876Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-dd689/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.245059047Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-dd689/insights-operator-5885fdb759-dd689.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24516163Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-dd689/insights-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.245170871Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-dd689/insights-operator/insights-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.245184421Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-dd689/insights-operator/insights-operator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.245231622Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-dd689/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.245681625Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-dd689/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246023715Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246048576Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/insights-runtime-extractor-5d2kg.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246156819Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24617156Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/exporter/exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24617785Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/exporter/exporter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24619244Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/exporter/exporter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246284743Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246353215Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/exporter/exporter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246377695Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/extractor/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246386436Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/extractor/extractor/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246392436Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/extractor/extractor/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246437207Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/extractor/extractor/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24653405Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246605322Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/extractor/extractor/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246619182Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246629373Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246637873Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246709035Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246803348Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24687405Z namespaces/openshift-insights/pods/insights-runtime-extractor-5d2kg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246939912Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.246981633Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/insights-runtime-extractor-8rxsh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247069515Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247079386Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/exporter/exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247094566Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/exporter/exporter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247136487Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/exporter/exporter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.2472292Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247291782Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/exporter/exporter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247393195Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/extractor/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247400785Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/extractor/extractor/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247404765Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/extractor/extractor/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247452506Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/extractor/extractor/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24755608Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247621911Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/extractor/extractor/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247663143Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247674153Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247678143Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247708994Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247799627Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24790951Z namespaces/openshift-insights/pods/insights-runtime-extractor-8rxsh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247934931Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.247953561Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/insights-runtime-extractor-rr5g7.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248050204Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248060054Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/exporter/exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248066694Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/exporter/exporter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248107796Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/exporter/exporter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248200288Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24827215Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/exporter/exporter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248306381Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/extractor/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248314742Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/extractor/extractor/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248326572Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/extractor/extractor/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248364083Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/extractor/extractor/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248460626Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248536498Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/extractor/extractor/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248551648Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248559349Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248567169Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24862393Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248711323Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248794255Z namespaces/openshift-insights/pods/insights-runtime-extractor-rr5g7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248827956Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.248851997Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24894324Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.24896755Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249040843Z namespaces/openshift-keda/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249050263Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249055263Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249080353Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249179807Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249228128Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249235718Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249264989Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249336741Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249380262Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249472255Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249555277Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249617829Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249625909Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249669761Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.249822565Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.250121804Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25034539Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.250533676Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25067821Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.250849185Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.250908917Z namespaces/openshift-keda/operators.coreos.com/installplans/install-qqkng.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25101398Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25104424Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251114563Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251140273Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251206265Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251254787Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251332109Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25137831Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251450342Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251483523Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251535244Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251578856Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251664518Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251743041Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251824543Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251872574Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251944436Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.251984828Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252014798Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252113251Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252157442Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252207334Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252292537Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252347048Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252390179Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252493102Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252568914Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252648687Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252737649Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252825362Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.252989927Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253179322Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253223864Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253251894Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253314716Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253350127Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253408209Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25344212Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253531482Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253612395Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253660826Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253706257Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253737208Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.2537917Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253830741Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253839481Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.253914873Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254057277Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254241343Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254378857Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254588723Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254778888Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254812789Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254875371Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254942613Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.254998465Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255037976Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255098278Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255128528Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255196241Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255228171Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255320534Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255397646Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255477118Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255509479Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255552381Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255604572Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255643173Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255732626Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25588807Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.255906911Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256002804Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256065475Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256112447Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25621551Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256296032Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256377205Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256452617Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256531229Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256668893Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256855898Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25689784Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256969692Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.256984342Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257044754Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257096596Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257114296Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257231429Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257305551Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257356543Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257393774Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257464246Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257486347Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25760001Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257630571Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257689702Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257725654Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257787755Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257824426Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257833127Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.257896148Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.258064003Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.258239718Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.258372752Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.258565578Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.258773754Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.258790354Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.258894157Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.258912568Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259037552Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259076753Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259144885Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259161045Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259253648Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259271918Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259376111Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259445303Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259525346Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259574697Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259613298Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.25967119Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259717621Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259805294Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259877616Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.259931827Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26002148Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260096952Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260133713Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260240476Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260319799Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260400091Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260480143Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260559485Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260687869Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.260945677Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261009469Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26104532Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261104881Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261138012Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261193014Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261243845Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261327578Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26140302Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261457971Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261479052Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261547034Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261592175Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261648377Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261689188Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26173347Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26174364Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261796891Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.261958356Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.262151342Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.262286175Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.262486161Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.262740799Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.262755179Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.262828441Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.262880843Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.262963995Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263012016Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263073578Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26311447Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263176841Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263204752Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263299075Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263374387Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263457109Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263507081Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263550492Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263604514Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263651615Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263731717Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263784779Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26382032Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263937783Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263958944Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.263965084Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.264020566Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.264079797Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.264110738Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.264219722Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.264297884Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.264705856Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.264797818Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.264879921Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265028215Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265190399Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26522468Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265281482Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265327163Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265370285Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265425816Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265474268Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26554974Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265630692Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265686544Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265703754Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265767916Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265806387Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265841008Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265847638Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.265915711Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.266067535Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.266254Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.266388884Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26660009Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.266780805Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.266807746Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.266885488Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26693687Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.266988991Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267034323Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267105215Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267146806Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267199088Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267229598Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267322921Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267399723Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267483546Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267531207Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267577388Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26762571Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267659041Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267750033Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267794295Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.267843446Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268001521Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268054522Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268087263Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268206157Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268285199Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268370662Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268445604Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268596248Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268717241Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268930398Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.268965369Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26901803Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269059441Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269114063Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269174545Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269226306Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269302078Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.26937722Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269422972Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269451753Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269511675Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269538915Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269593327Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269601227Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269631928Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269792712Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.269987138Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270125522Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270331148Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270523844Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270560545Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270628187Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270644787Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270716849Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270771071Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270830642Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270871104Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270938716Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.270987497Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271067309Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271131201Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271217354Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271265525Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271291406Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271360768Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271380829Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271488062Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271505702Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271569254Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271642196Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271695018Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271728939Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271835662Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.271933934Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272015437Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272097049Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272175241Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272300235Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27247027Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272521582Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272541252Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272609264Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272647335Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272707877Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272727168Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272852011Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272953954Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.272978005Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.273027936Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.273080198Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.273117179Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27315857Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.2731662Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.273209461Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.273358626Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.273536151Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.273673555Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.273919652Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274102597Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274120398Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274179899Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274225931Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274297293Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274341584Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274409316Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274424486Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274493368Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27453809Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274624722Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274735155Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274843169Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27490353Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.274947682Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275005413Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275044064Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275132617Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275184508Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275204629Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275305742Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275338533Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275348823Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275403315Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275477037Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275507428Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275632632Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275710004Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.275893179Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276036233Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276187617Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276311981Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276504117Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276570988Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276607629Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276651841Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276694212Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276769074Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276795205Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276919889Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.276997451Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.277047002Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.277098094Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.277153725Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.277200887Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.277243678Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.277260868Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27732576Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.277485225Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27764931Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.277785754Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27801431Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278217966Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278228396Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278263997Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278398471Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq/kube-storage-version-migrator-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278405901Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278417002Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278475914Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278585887Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-nkdqq/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27869932Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278737191Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278772282Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278827704Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278929267Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.278979808Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27903863Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279078851Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279137273Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279177374Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279266186Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27937314Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279468252Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279516184Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279554795Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279608906Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279641217Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.27973374Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279781061Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279814012Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279927116Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.279949296Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280017848Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280123031Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280193463Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280311027Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280387439Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280512033Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280634966Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280824991Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280875063Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280930705Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.280994306Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281023067Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281085509Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.28111718Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281206453Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281288955Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281335746Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281364867Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281424509Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.28145329Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281519471Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281529672Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281547032Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281780679Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.281991255Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.28215352Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282359376Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282548101Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282556061Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282615533Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/migrator-6865c9967-dftrh.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282709296Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/graceful-termination/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282717796Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/graceful-termination/graceful-termination/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282721636Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282775758Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282873631Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282969904Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282980874Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/migrator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282985754Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/migrator/migrator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.282994704Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/migrator/migrator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283078757Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/migrator/migrator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283191Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283257852Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-dftrh/migrator/migrator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283277422Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283338724Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283371675Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283410636Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283489179Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283519409Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283597662Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283618632Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283691404Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283757776Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.283844039Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284070535Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284191559Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.28422901Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284265951Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284327933Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284372654Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284440206Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284498538Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284549859Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284625101Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284672663Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284682223Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284717814Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284835138Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.28490692Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.284953651Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.285085505Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.285176107Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.285480286Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.285560098Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.285754304Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.285918369Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286121345Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286184417Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286223218Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286286909Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286343611Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286402743Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286466905Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286567268Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.28665096Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286707352Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286728132Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286812495Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286832775Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286935898Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.286959329Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.287027771Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.287038931Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.287083163Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.287237997Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.287409162Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.287548316Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.287799343Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288007829Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288189215Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288222916Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-dlwh8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288314308Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288332799Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288402601Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288470992Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288528544Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288579416Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288622817Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288642848Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-5zff2/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288702939Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-5zff2/lws-controller-manager-78dcd69c74-5zff2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288807422Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-5zff2/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288820223Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-5zff2/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288830993Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-5zff2/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288840783Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-5zff2/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.288986998Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-5zff2/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289058349Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-5zff2/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.28907613Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-c9ztf/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289137862Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-c9ztf/lws-controller-manager-78dcd69c74-c9ztf.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289213024Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-c9ztf/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289223534Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-c9ztf/manager/manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289229934Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-c9ztf/manager/manager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289239795Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-c9ztf/manager/manager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289503962Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-c9ztf/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289566074Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78dcd69c74-c9ztf/manager/manager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289613856Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dm88t/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289642777Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dm88t/openshift-lws-operator-fd8ccff4c-dm88t.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289735089Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dm88t/openshift-lws-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289747389Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dm88t/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.28975311Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dm88t/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.289819671Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dm88t/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290011367Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dm88t/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.29009689Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dm88t/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.29010981Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290155231Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290219303Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290233474Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290291935Z namespaces/openshift-machine-api/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290300716Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290305186Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290346247Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290502501Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290682796Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.290847351Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291086438Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291269564Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291279494Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291284594Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291309745Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291428448Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291633414Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291640924Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.291684605Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.29184659Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292107008Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292240161Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292444527Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292621723Z namespaces/openshift-marketplace/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292632073Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292637003Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292668434Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292734596Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292760957Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292814058Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.292927052Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.293014804Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.293094626Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.293140078Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.293173628Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.293342124Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.293516878Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.293653593Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.293877289Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.294078565Z namespaces/openshift-monitoring/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.294115366Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.294196548Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.294229389Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.294316842Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.294344993Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.294510927Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295005401Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295358722Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295588909Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295620929Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295674931Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295704992Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295803045Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295877287Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.295933798Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.296030361Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.296047582Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.296116414Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.299182762Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.299327037Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.300538521Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.300618884Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.301549811Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.301675394Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302182219Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302338274Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302356574Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302505118Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302522559Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302589021Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302608741Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302714615Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302790707Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302843888Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.302881709Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303046054Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303066195Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303163308Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303182008Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303283421Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303324452Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303438785Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3035933Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303680052Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.303978771Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304113955Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304245329Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304334551Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304447235Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304550677Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304619719Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304675671Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304765514Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.304874497Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30497543Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305073133Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305158415Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30531764Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305406122Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305502025Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305587127Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30567985Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305780003Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305844165Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305882516Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305950858Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.305958588Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.306002759Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.306197415Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.306406561Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.306550915Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.306753751Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307001479Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30703614Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307139012Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307148843Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307223325Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30739897Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30740798Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30741361Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307454992Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307556895Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307616376Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307660878Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307668538Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307672688Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307706429Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307809212Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307881084Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307990047Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.307999557Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308005017Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308039398Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308128651Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308211634Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308225814Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308231014Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308237554Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308285656Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308379488Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30845215Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308493412Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308502602Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308514732Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308550733Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308649826Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308716308Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308764059Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30877294Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30877723Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308819201Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.308933684Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309007766Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309027677Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309040038Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309045618Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309089699Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309179451Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309246613Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309276354Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-tg52h/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309333916Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-tg52h/cluster-monitoring-operator-548b489cb8-tg52h.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309424389Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-tg52h/cluster-monitoring-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309432679Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-tg52h/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.309437179Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-tg52h/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.30948055Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-tg52h/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310189981Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-tg52h/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310270503Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-tg52h/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310288454Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310346135Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-state-metrics-56d9f944b4-lzctb.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310453138Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-main/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310461288Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310465409Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31050621Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310605603Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310681145Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310714706Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-self/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310725956Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310730216Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310762967Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31085302Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310951413Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310972483Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-state-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310978583Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.310988994Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311034735Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311240211Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311313263Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-lzctb/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311332354Z namespaces/openshift-monitoring/pods/metrics-server-7bb87447bb-z4kjc/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311388435Z namespaces/openshift-monitoring/pods/metrics-server-7bb87447bb-z4kjc/metrics-server-7bb87447bb-z4kjc.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311481278Z namespaces/openshift-monitoring/pods/metrics-server-7bb87447bb-z4kjc/metrics-server/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311488668Z namespaces/openshift-monitoring/pods/metrics-server-7bb87447bb-z4kjc/metrics-server/metrics-server/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311492528Z namespaces/openshift-monitoring/pods/metrics-server-7bb87447bb-z4kjc/metrics-server/metrics-server/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311534139Z namespaces/openshift-monitoring/pods/metrics-server-7bb87447bb-z4kjc/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311631532Z namespaces/openshift-monitoring/pods/metrics-server-7bb87447bb-z4kjc/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311698854Z namespaces/openshift-monitoring/pods/metrics-server-7bb87447bb-z4kjc/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311736785Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-tg5vj/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311764486Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-tg5vj/monitoring-plugin-68b7fc9574-tg5vj.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311855229Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-tg5vj/monitoring-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31188566Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-tg5vj/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3118898Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-tg5vj/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.311924561Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-tg5vj/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312026394Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-tg5vj/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312098496Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-tg5vj/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312136347Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312180358Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/node-exporter-4zb96.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312287561Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/init-textfile/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312295821Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/init-textfile/init-textfile/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312299882Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/init-textfile/init-textfile/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312339473Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312414585Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312478737Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312512818Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312519678Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312528278Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31258811Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312680403Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312761865Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312808636Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/node-exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312815987Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/node-exporter/node-exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312822337Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/node-exporter/node-exporter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.312869938Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313009332Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313078864Z namespaces/openshift-monitoring/pods/node-exporter-4zb96/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313119445Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313171247Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/node-exporter-jrx9p.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31328653Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/init-textfile/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31329421Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/init-textfile/init-textfile/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313298381Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/init-textfile/init-textfile/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313343062Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313421884Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313569068Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313575229Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313579279Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313582979Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313586769Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313660141Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313726903Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313769904Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/node-exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313777424Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/node-exporter/node-exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313781455Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/node-exporter/node-exporter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.313819366Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31394964Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314022131Z namespaces/openshift-monitoring/pods/node-exporter-jrx9p/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314060623Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314115144Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/node-exporter-kn7nm.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314225517Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/init-textfile/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314236328Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/init-textfile/init-textfile/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314242708Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/init-textfile/init-textfile/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314274979Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314351391Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314421453Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314441094Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314449404Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314458934Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314510796Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314605218Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314677121Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314721472Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/node-exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314730952Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/node-exporter/node-exporter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314735292Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/node-exporter/node-exporter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314770333Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314881026Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.314968099Z namespaces/openshift-monitoring/pods/node-exporter-kn7nm/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31501119Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315038221Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/openshift-state-metrics-b8f49f5c6-6mt7w.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315186135Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-main/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315193275Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315197265Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315250247Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31534673Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315414862Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315444283Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-self/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315451313Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315460653Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315518565Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315605697Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315672039Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31571141Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/openshift-state-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315718371Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315727921Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315788062Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315889245Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.315968768Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-6mt7w/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31603687Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316113862Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316309848Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316317948Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316331728Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316371739Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316475533Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316542734Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316560925Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316572315Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316582045Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316652418Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316758441Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316824792Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316839963Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316846173Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316851733Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.316949326Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317037469Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317113101Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317140782Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317148962Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317153112Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317191783Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317280046Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317352028Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317393189Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317401269Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317405329Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317478562Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317569694Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317669117Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317705278Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317712248Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.317716138Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31776485Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318344287Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318417169Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318450139Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31845722Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31846175Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318510691Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318616264Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318683376Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318710937Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318771709Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/prometheus-operator-84d69f794d-vq8rs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318884302Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318897012Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318901333Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.318938694Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319039777Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319108169Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31914191Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/prometheus-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31914844Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31915509Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319202431Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319449988Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.31951632Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-vq8rs/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319545561Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-98g9k/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319597963Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-98g9k/prometheus-operator-admission-webhook-7f8d78f54b-98g9k.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319716416Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-98g9k/prometheus-operator-admission-webhook/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319723866Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-98g9k/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319727847Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-98g9k/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319777218Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-98g9k/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319889111Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-98g9k/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.319960693Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-98g9k/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320010095Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320050646Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/telemeter-client-5df795b4f-bxd4s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32019125Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32019959Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32020339Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320240431Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320333804Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320402036Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320448087Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/reload/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320456677Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/reload/reload/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320460638Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/reload/reload/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320494209Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/reload/reload/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320594261Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320663243Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/reload/reload/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320706685Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/telemeter-client/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320713735Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/telemeter-client/telemeter-client/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320719985Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320778777Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.320953752Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321025564Z namespaces/openshift-monitoring/pods/telemeter-client-5df795b4f-bxd4s/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321057835Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321124207Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/thanos-querier-b66997cdf-r8knr.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321276721Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321283921Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321287872Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321325453Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321419825Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321489618Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321522519Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-rules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321529459Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321535449Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32158107Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321676663Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321745385Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321782706Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-web/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321789216Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321793066Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321835697Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.321944381Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322020783Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322038133Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322044354Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322053164Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322109355Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322195158Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3222674Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32228148Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/prom-label-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32228933Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322295681Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322343012Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322433115Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322503027Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322550078Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/thanos-query/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322561478Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/thanos-query/thanos-query/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322566419Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/thanos-query/thanos-query/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322595179Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322703562Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322771535Z namespaces/openshift-monitoring/pods/thanos-querier-b66997cdf-r8knr/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322789565Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322844227Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.322898368Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32295506Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.323040732Z namespaces/openshift-multus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.323089654Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.323210007Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.323240888Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32329598Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.323324191Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32366649Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.323743153Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.323830865Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.323884447Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32398535Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324142934Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324273218Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3243354Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324365361Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324422272Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324477364Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324545856Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324583447Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324632598Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324719351Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324775553Z namespaces/openshift-multus/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324835784Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.324990249Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.325066191Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.325565955Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.325690909Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.326304697Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.326398029Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.326550474Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.326604555Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.326652237Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32675405Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.326836112Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.326954955Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327045628Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327094189Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327188922Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327277365Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327367947Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327419499Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32746784Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327520352Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327564733Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327692007Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327725278Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327780539Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327831191Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.327935654Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328015046Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328067777Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328116599Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.32816388Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328212902Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328260983Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328303444Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328353066Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328362446Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328416658Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328567592Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328755888Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.328913742Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.329131648Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.329320684Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.329335504Z namespaces/openshift-multus/pods/multus-9cmk4/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.329348374Z namespaces/openshift-multus/pods/multus-9cmk4/multus-9cmk4.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.329445817Z namespaces/openshift-multus/pods/multus-9cmk4/kube-multus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.329455348Z namespaces/openshift-multus/pods/multus-9cmk4/kube-multus/kube-multus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.329459538Z namespaces/openshift-multus/pods/multus-9cmk4/kube-multus/kube-multus/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.329495269Z namespaces/openshift-multus/pods/multus-9cmk4/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330132097Z namespaces/openshift-multus/pods/multus-9cmk4/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330199929Z namespaces/openshift-multus/pods/multus-9cmk4/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330254801Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330296272Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/multus-additional-cni-plugins-bgxrn.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330439196Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/bond-cni-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330448346Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330453106Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330490668Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3305849Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330647842Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330680723Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330686813Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/cni-plugins/cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330692213Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330745535Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330831888Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33091806Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330940101Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/egress-router-binary-copy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330951111Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.330991122Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331050434Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331133796Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331199928Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331239979Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3312505Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33125584Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33126249Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331352772Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331422854Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331446265Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/routeoverride-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331455375Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331465226Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331492616Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331588629Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331659381Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331686672Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni-bincopy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331694202Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331709163Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331747124Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.331837417Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332042512Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332079373Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332085694Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332090784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332134685Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332232938Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33230448Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgxrn/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332321951Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332389022Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/multus-additional-cni-plugins-shgn2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332528797Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/bond-cni-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332539507Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332545627Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332558887Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33265907Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332723252Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332792064Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332801134Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/cni-plugins/cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332805374Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332811965Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.332948959Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333035371Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333082062Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/egress-router-binary-copy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333092883Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333099363Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333148915Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333262258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333360671Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333393391Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333400872Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333412292Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333460113Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333554656Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333647619Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33368837Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/routeoverride-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33369571Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33369986Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333758002Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333855925Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333979808Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.333994999Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni-bincopy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334001629Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334010639Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334074451Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334170874Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334254936Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334301258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334311438Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334320258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33436943Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334477983Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334561625Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-shgn2/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334645438Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33470704Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/multus-additional-cni-plugins-z5plq.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334903485Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/bond-cni-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334928006Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334935026Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.334990548Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335088971Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335176343Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335229795Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335238295Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/cni-plugins/cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335244655Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335294637Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335431601Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335515873Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335551344Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/egress-router-binary-copy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335557974Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335561874Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335622606Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33575941Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335843102Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335906654Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335921685Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335927455Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.335982026Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336059229Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336145171Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336195283Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/routeoverride-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336203503Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336209713Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336248224Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336355937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33644002Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336490971Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni-bincopy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336499921Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336506581Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336552733Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336665396Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336736628Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33680187Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336814561Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336820601Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336886303Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.336997376Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.337075018Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-z5plq/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.337108829Z namespaces/openshift-multus/pods/multus-fxc56/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.337165661Z namespaces/openshift-multus/pods/multus-fxc56/multus-fxc56.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.337283314Z namespaces/openshift-multus/pods/multus-fxc56/kube-multus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.337295634Z namespaces/openshift-multus/pods/multus-fxc56/kube-multus/kube-multus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.337301635Z namespaces/openshift-multus/pods/multus-fxc56/kube-multus/kube-multus/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.337320065Z namespaces/openshift-multus/pods/multus-fxc56/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.338158879Z namespaces/openshift-multus/pods/multus-fxc56/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.338245862Z namespaces/openshift-multus/pods/multus-fxc56/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.338296313Z namespaces/openshift-multus/pods/multus-p94q8/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.338359045Z namespaces/openshift-multus/pods/multus-p94q8/multus-p94q8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.338467948Z namespaces/openshift-multus/pods/multus-p94q8/kube-multus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.338480879Z namespaces/openshift-multus/pods/multus-p94q8/kube-multus/kube-multus/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.338486629Z namespaces/openshift-multus/pods/multus-p94q8/kube-multus/kube-multus/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.33852537Z namespaces/openshift-multus/pods/multus-p94q8/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339379975Z namespaces/openshift-multus/pods/multus-p94q8/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339448537Z namespaces/openshift-multus/pods/multus-p94q8/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339488118Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339528339Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/network-metrics-daemon-2ksj4.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339616671Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339623942Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339628242Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339717785Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339815587Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339936881Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339960652Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/network-metrics-daemon/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339979702Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.339987722Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340039204Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340195048Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34026226Z namespaces/openshift-multus/pods/network-metrics-daemon-2ksj4/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340298331Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340339792Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/network-metrics-daemon-t5x5r.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340419285Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340429385Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340442135Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340449276Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340549778Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340618171Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340647261Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/network-metrics-daemon/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340655392Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340661752Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340715303Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340916449Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.340994181Z namespaces/openshift-multus/pods/network-metrics-daemon-t5x5r/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341025312Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341071404Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/network-metrics-daemon-vct9g.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341148376Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341158296Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341166306Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341188987Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34128548Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341349962Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341396463Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/network-metrics-daemon/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341406463Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341412534Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341448795Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341605029Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341674201Z namespaces/openshift-multus/pods/network-metrics-daemon-vct9g/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341703332Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341748983Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341801225Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341808225Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341842096Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.341959929Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342011931Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342059932Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342137975Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342190006Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342241318Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342286069Z namespaces/openshift-must-gather-6zt68/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342296959Z namespaces/openshift-must-gather-6zt68/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342303169Z namespaces/openshift-must-gather-6zt68/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3423434Z namespaces/openshift-must-gather-6zt68/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342550286Z namespaces/openshift-must-gather-6zt68/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342753492Z namespaces/openshift-must-gather-6zt68/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.342914287Z namespaces/openshift-must-gather-6zt68/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343149534Z namespaces/openshift-must-gather-6zt68/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343350319Z namespaces/openshift-network-console/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343394251Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343464183Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343491183Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343542235Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343593257Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343676239Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343787382Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343918316Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.343968057Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344017619Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344086971Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344132892Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344179534Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344210844Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344303147Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344351828Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34438822Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344472232Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344501633Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344552414Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344664888Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34475832Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344907635Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.344997617Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345152312Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345278975Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345396049Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34545038Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345499022Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345549583Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345596635Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345646406Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345693207Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345741399Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34578797Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345830951Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345877903Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.345984626Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346062928Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346110739Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34615014Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346201432Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346249903Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346292024Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346299325Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346336666Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34648689Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346663795Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.346800109Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347024636Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347221911Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347232772Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-sk5qv/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347261323Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-sk5qv/networking-console-plugin-74c9f884b7-sk5qv.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347344865Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-sk5qv/networking-console-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347366216Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-sk5qv/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347370356Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-sk5qv/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347379046Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-sk5qv/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347480449Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-sk5qv/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347550301Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-sk5qv/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347592462Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347629923Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347703925Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347744297Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347794308Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347839359Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347932662Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.347968833Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348019754Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348063966Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348173049Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348272102Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348365655Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348413256Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348458277Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348519699Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34856613Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348627372Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348671003Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348710074Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348760846Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348852779Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348941661Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.348983312Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.349069985Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34924001Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34924986Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.34925799Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.349336333Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.349735184Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.349810856Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350049643Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350137666Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35026597Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350321091Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350366082Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350424644Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350468615Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350568888Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350602459Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35064889Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350712442Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350767794Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350811385Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.350853776Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35096384Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351046602Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351096283Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351134325Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351189346Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351237728Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35131408Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351341221Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351366881Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351375131Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351412463Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351559527Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351735872Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.351976009Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352106613Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352304129Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352312329Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pd6zq/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35234548Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pd6zq/network-check-source-7d88cb55d8-pd6zq.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352428972Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pd6zq/check-endpoints/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352437772Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pd6zq/check-endpoints/check-endpoints/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352441813Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pd6zq/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352474323Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pd6zq/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352586607Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pd6zq/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352654248Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pd6zq/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35269703Z namespaces/openshift-network-diagnostics/pods/network-check-target-2psct/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352780752Z namespaces/openshift-network-diagnostics/pods/network-check-target-2psct/network-check-target-2psct.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352822664Z namespaces/openshift-network-diagnostics/pods/network-check-target-2psct/network-check-target-container/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352830184Z namespaces/openshift-network-diagnostics/pods/network-check-target-2psct/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352834204Z namespaces/openshift-network-diagnostics/pods/network-check-target-2psct/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352873495Z namespaces/openshift-network-diagnostics/pods/network-check-target-2psct/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.352991358Z namespaces/openshift-network-diagnostics/pods/network-check-target-2psct/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35305903Z namespaces/openshift-network-diagnostics/pods/network-check-target-2psct/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353095481Z namespaces/openshift-network-diagnostics/pods/network-check-target-54qd4/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353143253Z namespaces/openshift-network-diagnostics/pods/network-check-target-54qd4/network-check-target-54qd4.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353211225Z namespaces/openshift-network-diagnostics/pods/network-check-target-54qd4/network-check-target-container/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353218685Z namespaces/openshift-network-diagnostics/pods/network-check-target-54qd4/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353223025Z namespaces/openshift-network-diagnostics/pods/network-check-target-54qd4/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353257166Z namespaces/openshift-network-diagnostics/pods/network-check-target-54qd4/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353347579Z namespaces/openshift-network-diagnostics/pods/network-check-target-54qd4/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353426741Z namespaces/openshift-network-diagnostics/pods/network-check-target-54qd4/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353452982Z namespaces/openshift-network-diagnostics/pods/network-check-target-82qzb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353497023Z namespaces/openshift-network-diagnostics/pods/network-check-target-82qzb/network-check-target-82qzb.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353574725Z namespaces/openshift-network-diagnostics/pods/network-check-target-82qzb/network-check-target-container/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353583166Z namespaces/openshift-network-diagnostics/pods/network-check-target-82qzb/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353587346Z namespaces/openshift-network-diagnostics/pods/network-check-target-82qzb/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353619216Z namespaces/openshift-network-diagnostics/pods/network-check-target-82qzb/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353767271Z namespaces/openshift-network-diagnostics/pods/network-check-target-82qzb/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353883844Z namespaces/openshift-network-diagnostics/pods/network-check-target-82qzb/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353892584Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353902675Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353913435Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353928765Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.353977297Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354063529Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354109871Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354152132Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354237644Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354274146Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354322317Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354378678Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35443214Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354497992Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354537063Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354585284Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354632656Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354710728Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354796141Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354873783Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354961765Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.354980826Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355044918Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355074958Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355159341Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355209313Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355235893Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355327676Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355375117Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355381967Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355434669Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35547386Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355516951Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355625854Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355700997Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355789469Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.355871332Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356049397Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356227442Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356330385Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356361916Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356425358Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356465389Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35650858Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356559141Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356588282Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356645004Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356672685Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356777108Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356874671Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.356959243Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.357007885Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.357071936Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.357110908Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.357164989Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.357172359Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35721029Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.357402666Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.357923481Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358104816Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358326693Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358542679Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.35857638Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358627991Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358637332Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358677523Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358730704Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358775096Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358832417Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358889229Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.358961651Z namespaces/openshift-network-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359000802Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359090285Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359128876Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359177137Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359218898Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359328501Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359403974Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359488426Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359530627Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359559908Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359637021Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359677442Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359723483Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359768184Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359853267Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.359916998Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3599586Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360049702Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360095674Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360102494Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360144825Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360198707Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360250358Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360376752Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360472385Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360633299Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360709661Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.360917978Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36100807Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361157614Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361224847Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361276368Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36136037Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361419462Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361481834Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361537185Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361572377Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361627458Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361670439Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36170945Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361750492Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361844054Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361931067Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.361979778Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36202079Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362075141Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362131563Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362177824Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362211505Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362267157Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362303878Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362362149Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362371209Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362402581Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362559755Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36274271Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.362886024Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363189833Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363373529Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363381749Z namespaces/openshift-network-operator/pods/iptables-alerter-c4d6f/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36341801Z namespaces/openshift-network-operator/pods/iptables-alerter-c4d6f/iptables-alerter-c4d6f.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363663117Z namespaces/openshift-network-operator/pods/iptables-alerter-c4d6f/iptables-alerter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363671917Z namespaces/openshift-network-operator/pods/iptables-alerter-c4d6f/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363675867Z namespaces/openshift-network-operator/pods/iptables-alerter-c4d6f/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363717948Z namespaces/openshift-network-operator/pods/iptables-alerter-c4d6f/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363816871Z namespaces/openshift-network-operator/pods/iptables-alerter-c4d6f/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363909014Z namespaces/openshift-network-operator/pods/iptables-alerter-c4d6f/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363940525Z namespaces/openshift-network-operator/pods/iptables-alerter-cqgbv/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.363988936Z namespaces/openshift-network-operator/pods/iptables-alerter-cqgbv/iptables-alerter-cqgbv.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364060178Z namespaces/openshift-network-operator/pods/iptables-alerter-cqgbv/iptables-alerter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364067179Z namespaces/openshift-network-operator/pods/iptables-alerter-cqgbv/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364072449Z namespaces/openshift-network-operator/pods/iptables-alerter-cqgbv/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364157861Z namespaces/openshift-network-operator/pods/iptables-alerter-cqgbv/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364215393Z namespaces/openshift-network-operator/pods/iptables-alerter-cqgbv/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364284645Z namespaces/openshift-network-operator/pods/iptables-alerter-cqgbv/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364301765Z namespaces/openshift-network-operator/pods/iptables-alerter-gjsxw/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364359377Z namespaces/openshift-network-operator/pods/iptables-alerter-gjsxw/iptables-alerter-gjsxw.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364427489Z namespaces/openshift-network-operator/pods/iptables-alerter-gjsxw/iptables-alerter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364434889Z namespaces/openshift-network-operator/pods/iptables-alerter-gjsxw/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364440109Z namespaces/openshift-network-operator/pods/iptables-alerter-gjsxw/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36447395Z namespaces/openshift-network-operator/pods/iptables-alerter-gjsxw/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364564743Z namespaces/openshift-network-operator/pods/iptables-alerter-gjsxw/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364630765Z namespaces/openshift-network-operator/pods/iptables-alerter-gjsxw/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364671226Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364707867Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364770099Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36480897Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364900573Z namespaces/openshift-node/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364934714Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364939824Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.364948934Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.365107649Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.365263363Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.365395337Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.365613073Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.365796999Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36584015Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.365923422Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.365957313Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366012045Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366056616Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366146829Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366224131Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366308363Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366342495Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366388936Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366424397Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366481268Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366560961Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366598092Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366644313Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366722045Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366775417Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366840499Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.366938332Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367027714Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367123657Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367198849Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367286022Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367410725Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36759188Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367664983Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367713484Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367762196Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367808087Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.367854778Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36792151Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368002183Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368080985Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368131876Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368167697Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368216819Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368244659Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368313921Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368361023Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368439495Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368449975Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368464566Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36862469Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368818776Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.368977501Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369182266Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369450584Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369468395Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369534377Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369569128Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36963518Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.36965225Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369714412Z namespaces/openshift-operators/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369723662Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369728612Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369761533Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.369935168Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370116264Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370255848Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370464094Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370645889Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370664059Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370738422Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370787723Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370852925Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370973328Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.370996509Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371070721Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371247306Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371322899Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371403241Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371453182Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371498444Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371624997Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371662318Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37171444Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371753001Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371841754Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371901985Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.371938006Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.372023829Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37205695Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37206342Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.372113481Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.372169693Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.372215994Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37241196Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.372496312Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.372877273Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.373014257Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.373490011Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.373575444Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.373733068Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37378472Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37381668Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.373942444Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374057258Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374089699Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374177261Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374223222Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374272774Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374335846Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374362896Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374438618Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37447533Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374527781Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374535031Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374565612Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374622134Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374650205Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374752368Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37482441Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374857721Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374925553Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.374987394Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375025335Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375102578Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375146829Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37519734Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375205211Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375244772Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375329484Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375385066Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375424567Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375472928Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375479979Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37551802Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375666764Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.375844709Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376002344Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.37621753Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376404135Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376412406Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376481037Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovnkube-node-grsjn.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376640462Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-node/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376649162Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376653362Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376688323Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376891889Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376982332Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.376999623Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377007593Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377012303Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377061594Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377209138Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377276251Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377303161Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/nbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377310152Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/nbdb/nbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377314212Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/nbdb/nbdb/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377358873Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/nbdb/nbdb/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377456716Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377524308Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377562039Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/northd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377571449Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/northd/northd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377575639Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/northd/northd/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3776076Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/northd/northd/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377715293Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377782335Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/northd/northd/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377808116Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-acl-logging/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377814976Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377821466Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377874748Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.377982081Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378047803Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378084624Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378093114Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-controller/ovn-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378097464Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378129575Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378491976Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378557508Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378571638Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovnkube-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378584128Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378621419Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.378672701Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380420161Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380489073Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380513564Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/sbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380522374Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/sbdb/sbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380536205Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/sbdb/sbdb/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380572006Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/sbdb/sbdb/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380674049Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380750491Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grsjn/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380785262Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.380880925Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovnkube-node-gzpfl.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381030239Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-node/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381040369Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381045329Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.38107584Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381260986Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381335128Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381376789Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381384599Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381388779Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.38142416Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381580235Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381650207Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381686268Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/nbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381693388Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/nbdb/nbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381697358Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/nbdb/nbdb/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.38173886Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/nbdb/nbdb/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381831842Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381926145Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381944066Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/northd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381949676Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/northd/northd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.381956066Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/northd/northd/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382007127Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/northd/northd/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.38211489Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382183102Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/northd/northd/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382215103Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-acl-logging/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382221914Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382225843Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382261415Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382356987Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382425259Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.38244276Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382469781Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-controller/ovn-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382474841Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382515502Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.38278014Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382836291Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382895663Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovnkube-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382909293Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382914534Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.382951944Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.384664584Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.384764937Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.384811538Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/sbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.384820569Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/sbdb/sbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.384825519Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/sbdb/sbdb/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.38485594Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/sbdb/sbdb/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.384979713Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385052945Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gzpfl/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385073936Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385161639Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovnkube-node-jvw95.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385313393Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-node/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385320503Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385324463Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385350154Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385510229Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3855648Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385613181Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385620892Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385624842Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385653213Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385805057Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385875529Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385928611Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/nbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385936531Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/nbdb/nbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385940431Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/nbdb/nbdb/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.385982112Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/nbdb/nbdb/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386077225Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386153227Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386163968Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/northd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386175568Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/northd/northd/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386186518Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/northd/northd/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386224779Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/northd/northd/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386343353Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386410775Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/northd/northd/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386449836Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-acl-logging/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386459966Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386463916Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386473097Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3865778Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386651731Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386668632Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386675082Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-controller/ovn-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386679062Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.386728884Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.387117035Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.387180487Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.387211598Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovnkube-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.387219268Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.387223248Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.387247729Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.388949598Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.38901542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389036431Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/sbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389045341Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/sbdb/sbdb/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389053691Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/sbdb/sbdb/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389096642Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/sbdb/sbdb/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389199285Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389265067Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jvw95/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389285628Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389334919Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389392891Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389402741Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389410931Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389524545Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389648108Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389682829Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389734211Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389830854Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389935766Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.389993958Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390014629Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39006756Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390079741Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390087281Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390140223Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390193914Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390200984Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390238115Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390429521Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390639087Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.390801011Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391130041Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391336737Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391369038Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39144225Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391470411Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391530933Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391559264Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391657166Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391759859Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391876603Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391917844Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.391972256Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392018277Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392055438Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39213595Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392184752Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392218432Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392305825Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392350136Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392358157Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392397658Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392445179Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392496701Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392622314Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392705157Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.392927283Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393013875Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393127409Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393245002Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393435388Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393477349Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39352893Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393580402Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393612773Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393666805Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393716116Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393788298Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393876571Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393937282Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.393975073Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394032275Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394075366Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394129778Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394171249Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39421945Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394226501Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394263842Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394410496Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394589331Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394728995Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.394950842Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395132587Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395140197Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-htbcm/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395175288Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-htbcm/service-ca-operator-5d6c75f84f-htbcm.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39526086Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-htbcm/service-ca-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395270811Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-htbcm/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395277311Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-htbcm/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395305032Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-htbcm/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395480677Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-htbcm/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395647982Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395690793Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395739594Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395778666Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395826627Z namespaces/openshift-service-ca/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.395877548Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396012582Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396051214Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396100575Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396143076Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396225988Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396323751Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396418854Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396466306Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396509057Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3966441Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396680302Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396778104Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396822526Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.396882518Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39697149Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397017061Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397024782Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397063303Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397114284Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397167766Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397264698Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397354691Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397469024Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397541626Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39765466Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397808444Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.397967809Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39801903Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398063692Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398108463Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398143914Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398194865Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398222116Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398316299Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398393121Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398441623Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398485104Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398530105Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398562836Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398615957Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398622778Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398664499Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.398814803Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.399068131Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.399202274Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3994103Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.399601616Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.399610746Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-xqpb8/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.399627457Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-xqpb8/service-ca-658d7b7c78-xqpb8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.399718949Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-xqpb8/service-ca-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.39972854Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-xqpb8/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.3997334Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-xqpb8/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.399763391Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-xqpb8/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400008448Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-xqpb8/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40007624Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-xqpb8/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400113901Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400152952Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400210774Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400230944Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400282146Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400333607Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400394299Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40043229Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400506252Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400524133Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400621226Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400678527Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40078786Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400816501Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400880293Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400947645Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.400963685Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401068788Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401089719Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401143321Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401221973Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401277014Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401308406Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401422909Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401498651Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401586994Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401663416Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401744338Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.401878132Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402083538Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402116529Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40216719Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402213642Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402259253Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402307224Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402349145Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402434698Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40251064Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402562002Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402602023Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402646254Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402692886Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402738637Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402746207Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402787338Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.402965363Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.403227571Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.403366555Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40356082Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.403743146Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.403786397Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.403825198Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40388162Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.403942922Z namespaces/openshift/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.403995343Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404042865Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404086986Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404164908Z namespaces/openshift/apps/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404205059Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404290372Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404377404Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404449446Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404495678Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404541039Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40458864Z namespaces/openshift/batch/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404628722Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404714004Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404765986Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404817837Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40492113Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.404968821Z namespaces/openshift/core/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405010073Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405120606Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405201538Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405291711Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405367703Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405452335Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405580979Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405754954Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405794705Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405838866Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405907779Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.405929279Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.407298099Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40733518Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.407437283Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.407518055Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.407627888Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.407727951Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.407812723Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.407983408Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.408147833Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.408270297Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.408365919Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.408461562Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.408586216Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.40871347Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.408844793Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.408961567Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409047699Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409161293Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409275026Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409392089Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409502572Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409592355Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409705828Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409793581Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.409921935Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.410039838Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.410146121Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.410257324Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.410410189Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.410528662Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.410646665Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.410768819Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.410921553Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.411031387Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.41115613Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.411261913Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.411400337Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.411589783Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.411701066Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.41183774Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.411969454Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.412087557Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.412219041Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.412314144Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.412406626Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.41254311Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.412677124Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.412801748Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.412933602Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.413028774Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.413125557Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.41322372Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.413346403Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.413473887Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.413591481Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.413700404Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.413803737Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.4139309Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414028273Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414147697Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414223349Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414282441Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414360723Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414440825Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414484627Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414531418Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414577769Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414620571Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414668162Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414675872Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414717893Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.414882668Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415114425Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415253029Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415470105Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.4156529Z namespaces/openshift/policy/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415694871Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415742063Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415785364Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415831926Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415841056Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.415890327Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.416019851Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.416137254Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.416243387Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.416356081Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.416534406Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.416638229Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.416764273Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.416914787Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.417042701Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.417155704Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.417269167Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.417396821Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.417541115Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.417643688Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.417762811Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.417881585Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.418172523Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.418285737Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.418381259Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.418494283Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.418601795Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.418706929Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.418802611Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.418926935Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.419036628Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.419133711Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.419232274Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.419371328Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.419543323Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.419678767Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.419777029Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.419894533Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.420035457Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.42015244Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.420252383Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.420354236Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.420452279Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.420569452Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.420681496Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.420803609Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.420950154Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.421066307Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.42119067Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.421314294Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.421440418Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.421587822Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.421716206Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.42185349Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.422004704Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.422094947Z network_logs/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.422140478Z network_logs/cluster_scale [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.42223252Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.422306423Z network_logs/multi-networkpolicy [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.422386035Z network_logs/net-attach-def [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.422473667Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.422640682Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.422719805Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.425085173Z nodes/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.425130914Z nodes/debug [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.425222177Z nodes/ip-10-0-128-242.ec2.internal/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.425251848Z nodes/ip-10-0-128-242.ec2.internal/cpu_affinities.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.425556457Z nodes/ip-10-0-128-242.ec2.internal/dmesg [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.425838995Z nodes/ip-10-0-128-242.ec2.internal/ethtool_channels [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.425949628Z nodes/ip-10-0-128-242.ec2.internal/ethtool_features [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.426040701Z nodes/ip-10-0-128-242.ec2.internal/ip-10-0-128-242.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.426480763Z nodes/ip-10-0-128-242.ec2.internal/irq_affinities.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.426617467Z nodes/ip-10-0-128-242.ec2.internal/lscpu [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.426754031Z nodes/ip-10-0-128-242.ec2.internal/lspci [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.426850464Z nodes/ip-10-0-128-242.ec2.internal/podresources.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.426982668Z nodes/ip-10-0-128-242.ec2.internal/pods_info.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.427109881Z nodes/ip-10-0-128-242.ec2.internal/proc_cmdline [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.427207534Z nodes/ip-10-0-128-242.ec2.internal/sysinfo.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.428946074Z nodes/ip-10-0-128-242.ec2.internal/sysinfo.tgz [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.429105919Z nodes/ip-10-0-131-193.ec2.internal/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.42913497Z nodes/ip-10-0-131-193.ec2.internal/cpu_affinities.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.429455339Z nodes/ip-10-0-131-193.ec2.internal/dmesg [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.429704847Z nodes/ip-10-0-131-193.ec2.internal/ethtool_channels [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.429781479Z nodes/ip-10-0-131-193.ec2.internal/ethtool_features [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.429888372Z nodes/ip-10-0-131-193.ec2.internal/ip-10-0-131-193.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.430342875Z nodes/ip-10-0-131-193.ec2.internal/irq_affinities.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.43050512Z nodes/ip-10-0-131-193.ec2.internal/lscpu [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.430635103Z nodes/ip-10-0-131-193.ec2.internal/lspci [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.430724046Z nodes/ip-10-0-131-193.ec2.internal/podresources.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.430837639Z nodes/ip-10-0-131-193.ec2.internal/pods_info.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.431048355Z nodes/ip-10-0-131-193.ec2.internal/proc_cmdline [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.431131868Z nodes/ip-10-0-131-193.ec2.internal/sysinfo.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.432962421Z nodes/ip-10-0-131-193.ec2.internal/sysinfo.tgz [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.433143816Z nodes/ip-10-0-135-98.ec2.internal/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.433163616Z nodes/ip-10-0-135-98.ec2.internal/cpu_affinities.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.433455125Z nodes/ip-10-0-135-98.ec2.internal/dmesg [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.433710322Z nodes/ip-10-0-135-98.ec2.internal/ethtool_channels [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.433785674Z nodes/ip-10-0-135-98.ec2.internal/ethtool_features [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.433892578Z nodes/ip-10-0-135-98.ec2.internal/ip-10-0-135-98.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.43432558Z nodes/ip-10-0-135-98.ec2.internal/irq_affinities.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.434478174Z nodes/ip-10-0-135-98.ec2.internal/lscpu [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.434611628Z nodes/ip-10-0-135-98.ec2.internal/lspci [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.434697801Z nodes/ip-10-0-135-98.ec2.internal/podresources.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.434805254Z nodes/ip-10-0-135-98.ec2.internal/pods_info.json [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.434938968Z nodes/ip-10-0-135-98.ec2.internal/proc_cmdline [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.43502233Z nodes/ip-10-0-135-98.ec2.internal/sysinfo.log [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.436699309Z nodes/ip-10-0-135-98.ec2.internal/sysinfo.tgz [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.436889524Z pod_network_connectivity_check/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.436918395Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.437046599Z static-pods/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.437056269Z static-pods/kube-apiserver/ [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.442894488Z [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.442921669Z sent 72,279 bytes received 8,268,197 bytes 5,560,317.33 bytes/sec [must-gather] [must-gather-qz5r8] OUT 2026-07-30T00:15:10.442932429Z total size is 102,722,440 speedup is 12.32 [must-gather] [must-gather ] OUT 2026-07-30T00:15:10.753190361Z namespace/openshift-must-gather-6zt68 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: f0d9d026-ee03-40d3-bb76-cba3903e5b76 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-m2fr7 [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-m2fr7-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-m2fr7 [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 16378178 Jul 30 00:17 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-m2fr7/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 33a16c0] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-m2fr7/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] d90df53..33a16c0 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-30T00:17:31.430Z","type":3}]