task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-nh8vt-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-nh8vt-kubeconfig [get-kubeconfig] Found admin password secret: cluster-nh8vt-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-nh8vt-password [get-kubeconfig] API Server URL: https://ad13b180d2555444a88e0c68148704cd-01ec408b3e1c96fe.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.55c0f14d-689e-464a-b498-f413cbf2de28.prod.konfluxeaas.com [clone-repo] work/RHOAIENG-79440 [clone-repo] https://github.com/jlost/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 46% (1500/3251) Updating files: 47% (1528/3251) Updating files: 48% (1561/3251) Updating files: 49% (1593/3251) Updating files: 50% (1626/3251) Updating files: 51% (1659/3251) Updating files: 52% (1691/3251) Updating files: 53% (1724/3251) Updating files: 54% (1756/3251) Updating files: 55% (1789/3251) Updating files: 56% (1821/3251) Updating files: 57% (1854/3251) Updating files: 58% (1886/3251) Updating files: 59% (1919/3251) Updating files: 60% (1951/3251) Updating files: 61% (1984/3251) Updating files: 62% (2016/3251) Updating files: 63% (2049/3251) Updating files: 64% (2081/3251) Updating files: 65% (2114/3251) Updating files: 66% (2146/3251) Updating files: 67% (2179/3251) Updating files: 68% (2211/3251) Updating files: 69% (2244/3251) Updating files: 70% (2276/3251) Updating files: 71% (2309/3251) Updating files: 72% (2341/3251) Updating files: 73% (2374/3251) Updating files: 74% (2406/3251) Updating files: 75% (2439/3251) Updating files: 76% (2471/3251) Updating files: 77% (2504/3251) Updating files: 78% (2536/3251) Updating files: 79% (2569/3251) Updating files: 80% (2601/3251) Updating files: 81% (2634/3251) Updating files: 82% (2666/3251) Updating files: 83% (2699/3251) Updating files: 84% (2731/3251) Updating files: 85% (2764/3251) Updating files: 86% (2796/3251) Updating files: 87% (2829/3251) Updating files: 88% (2861/3251) Updating files: 89% (2894/3251) Updating files: 90% (2926/3251) Updating files: 91% (2959/3251) Updating files: 92% (2991/3251) Updating files: 93% (3024/3251) Updating files: 94% (3056/3251) Updating files: 95% (3089/3251) Updating files: 96% (3121/3251) Updating files: 97% (3154/3251) Updating files: 98% (3186/3251) Updating files: 99% (3219/3251) Updating files: 100% (3251/3251) Updating files: 100% (3251/3251), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1821 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1821 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.85s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 427ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 49ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://ad13b180d2555444a88e0c68148704cd-01ec408b3e1c96fe.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: InstallReady). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-gjxrd 1/1 Running 0 40s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-gjxrd condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-8wdmv 1/1 Running 0 44s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-8wdmv condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-j42jn 1/1 Running 0 50s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-j42jn condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-545d98d4c7-5zmwp 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-545d98d4c7-5zmwp condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (20/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (20/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (25/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (30/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (35/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-z4vbw 1/1 Running 0 12s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-z4vbw condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-82lr6 0/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-82lr6 condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-8699b59bbc-pfnkg 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-r8p5c 1/1 Running 0 65s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-r8p5c condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-r8p5c 1/1 Running 0 75s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-r8p5c condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-nh8vt-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-nh8vt-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-nh8vt-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-nh8vt-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-nh8vt-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-nh8vt-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Tue Jul 28 17:33:17 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-z8vbf condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-cd4d44c67-4n9rn" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-cd4d44c67-tftvf condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:b5c835d640b650f164ba7f36d7008760c88e2974efa65853ee167323c441580c [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:77c0c8e0b30f40efcf2f0417b910a58f8b07503b5e75a61bdb9ab9efab31d46f [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.124.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'metadata': '3.1.1', 'cov': '5.0.0', 'json-report': '1.5.0', 'xdist': '3.6.1', 'asyncio': '0.23.8', 'httpx': '0.30.0', 'anyio': '4.9.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: metadata-3.1.1, cov-5.0.0, json-report-1.5.0, xdist-3.6.1, asyncio-0.23.8, httpx-0.30.0, anyio-4.9.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] 2026-07-28 17:35:44.743 6599 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-28 17:35:44.744 6596 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-28 17:38:21.296 6599 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 17:38:21.296 6599 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 17:38:21.324 6599 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:38:21.324 6599 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:38:21.334 6599 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:38:21.334 6599 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-28 17:38:31.879 6599 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 17:38:31.879 6599 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 17:38:31.906 6599 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:38:31.906 6599 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:38:31.914 6599 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:38:31.914 6599 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T17:48:12.005325', start_time = 1785260892.0057046 [e2e-llm-inference-service] duration = 900.6812553405762, timestamp_end = '2026-07-28T18:03:12.686971' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f5ad3ca5580> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-28T17:48:11.913339] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T17:48:11.926106] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T17:48:12.005094] end - ✅ in 0.079s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T17:48:12.005325] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:48:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:03:12.686971] end - ❌ 900.681s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:03:12.687204] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-x4wxb', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-v2lwq'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:03:32.818240] end - ✅ in 20.130s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-28T18:03:32.818419] end - ❌ 920.905s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:34Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_deployment(test_case: TestCase): [e2e-llm-inference-service] """KEDA + Deployment: ScaledObject exists with WVA annotations; no HPA; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:606: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:03:33.682220', start_time = 1785261813.6824994 [e2e-llm-inference-service] duration = 900.2734687328339, timestamp_end = '2026-07-28T18:18:33.955970' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f5ad39d51c0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_deployment] [2026-07-28T18:03:33.597054] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:03:33.610179] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:03:33.682104] end - ✅ in 0.072s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:03:33.682220] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:03:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'message': 'ScaledObject conditions not yet available', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:18:33.955970] end - ❌ 900.273s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:18:33.956050] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-77fc6d7cf9-vsc52', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-kkr57'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:18:59.272374] end - ✅ in 25.316s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_deployment] [2026-07-28T18:18:59.272566] end - ❌ 925.675s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:43Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:48Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:08:27.760039', start_time = 1785262107.7603772 [e2e-llm-inference-service] duration = 900.7581856250763, timestamp_end = '2026-07-28T18:23:28.518565' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f384d886b60> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:11:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-48639af5 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-28T18:08:27.492431] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-48639af5', before_test=[ at 0x7f384e11bc40>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:08:27.505080] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:08:27.759874] end - ✅ in 0.255s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:08:27.760039] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'message': 'Inference Pool e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:36:18Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:11:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 8: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:36:18Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:36:18Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:11:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:23:28.518565] end - ❌ 900.758s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:11:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:11:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'e2e-test-llm-inference-service-48639af5' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:27Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:27Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:27Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:11:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] resourceVersion: '58733' [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ab92baa5a231647a39d5ae6d48e6a3a5-1959185142.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ab92baa5a231647a39d5ae6d48e6a3a5-1959185142.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ab92baa5a231647a39d5ae6d48e6a3a5-1959185142.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:50Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:50Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:50Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:11:24Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:38Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:50Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:50Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ab92baa5a231647a39d5ae6d48e6a3a5-1959185142.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs to ip-10-0-135-177.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:02 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:05 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.134.0.70/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:05 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-177.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:05 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-177.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:05 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-177.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:07:57 e2e-test-llm-inference-service-48639af5 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-rwbfs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:24 e2e-test-llm-inference-service-48639af5 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:07:57 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:07:57 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:07:57 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-6db8dfcb6f-lznvq_877f7605-faee-4da5-93c7-57a298b82f9f Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-48639af5/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:07:59 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-6db8dfcb6f-lznvq_877f7605-faee-4da5-93c7-57a298b82f9f Normal ProvisioningSucceeded Successfully provisioned volume pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8 to ip-10-0-135-177.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:38 e2e-test-llm-inference-service-48639af5 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f to ip-10-0-133-142.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:38 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:45 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:45 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-142.ec2.internal Normal Pulling Pulling image "vllm/vllm-openai-cpu:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:09:14 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-142.ec2.internal Normal Pulled Successfully pulled image "vllm/vllm-openai-cpu:v0.19.0" in 28.853s (28.853s including waiting). Image size: 3531178100 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:09:14 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-142.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:09:14 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-142.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:11:05 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-133-142.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.48:8000/health": dial tcp 10.132.0.48:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:32 e2e-test-llm-inference-service-48639af5 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-132820f03254470cd8a0146008974a5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:54 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:18:39 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:49 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:36 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr to ip-10-0-135-177.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:37 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.134.0.72/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-177.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-177.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 18:08:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-135-177.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ec06b310-2da4-41a4-b84d-a06fbd9920fe', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 18:23:28 GMT', 'Content-Length': '262'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"llm-d-routing-sidecar\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '1b50ba6e-6937-4763-b3b4-a66ba0d6c00a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 18:23:28 GMT', 'Content-Length': '245'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:05 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:15 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:15 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:15 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:15 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:15 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:15 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:15 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 18:11:15 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:11:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:12:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:13:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:14:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:15:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:16:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:17:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:18:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:19:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:20:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:21:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:22:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 18:23:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785262117.3487642,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3490381,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"http","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"e2e-test-llm-inference-service-48639af5","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785262117.349359,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3514707,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3515186,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3519514,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3739924,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3740673,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3741546,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3741887,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.374225,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3742852,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.374305,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3743694,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3744485,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3744717,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3745167,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3745282,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3745508,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.374681,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3746924,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00032d7c0 DataConfig:{Sources:[{Plugin:0xc000338d80 Extractors:[0xc00032da80]}]} FlowControlConfig: ParserRegistry:0xc0005a1b80}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3952978,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3953261,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.395347,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3953545,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3953621,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.39537,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3953764,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.395384,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3953922,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3954017,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.39541,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.395419,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3954315,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3954425,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3954816,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785262117.395494,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3955293,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3955553,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3955698,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.395668,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785262117.395679,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.395716,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.39587,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3962033,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3962092,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3962653,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3962653,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3967972,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3970423,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3977664,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.3978693,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785262117.4055026,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.406218,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.4063358,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.4066453,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.4976432,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.4976742,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785262117.497709,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.4977376,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785262117.5979502,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.5979462,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785262117.5979934,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785262117.5979986,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785262117.5981586,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"60083ed5-0c87-4000-a44f-ccb18d3e717b"} [e2e-llm-inference-service] {"level":"info","ts":1785262129.161124,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"00137585-c1a8-4147-a250-828657dad77f"} [e2e-llm-inference-service] {"level":"info","ts":1785262275.5506897,"caller":"inflightload/producer.go:308","msg":"Injected dynamic attribute into endpoint","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f-rank-0"} [e2e-llm-inference-service] {"level":"info","ts":1785262275.550735,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f","reconcileID":"b0d05d9e-3ce9-4790-bbbe-9b29f22e115c"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 156c4f4e-944f-4916-98b9-39960bbfd616 [e2e-llm-inference-service] resourceVersion: '57326' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.72 [e2e-llm-inference-service] nodeName: ip-10-0-135-177.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr [e2e-llm-inference-service] uid: a175d464-b6e6-41ee-a9d8-c5d400230273 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a3d682a5-4c00-4489-b25c-0eb3627c6eeb [e2e-llm-inference-service] resourceVersion: '58639' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.48 [e2e-llm-inference-service] nodeName: ip-10-0-133-142.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f [e2e-llm-inference-service] uid: e40a0463-3c22-4106-a788-24bb2ce11bfc [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 1c409c83-3d4a-40a6-9d31-4982e37030c4 [e2e-llm-inference-service] resourceVersion: '56755' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.71/23"],"mac_address":"0a:58:0a:86:00:47","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.71/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] uid: 9c651a42-2db4-4a23-8fbb-50ade303c305 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-177 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9c651a42-2db4-4a23-8fbb-50ade303c305"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-l4t8s [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-l4t8s [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-l4t8s [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-135-177.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-xjjgk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2x7tv [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [llm-d-routing-sidecar]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] hostIP: 10.0.135.177 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.177 [e2e-llm-inference-service] startTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-l4t8s [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-l4t8s [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e40a0463-3c22-4106-a788-24bb2ce11bfc [e2e-llm-inference-service] resourceVersion: '58638' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.48/23"],"mac_address":"0a:58:0a:84:00:30","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.48/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.48\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:30\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] uid: 580a158a-ef00-44a6-9d08-ecd92b5b19ce [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-133-142 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"580a158a-ef00-44a6-9d08-ecd92b5b19ce"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.48"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-f66n2 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-f66n2 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-133-142.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-xjjgk [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:09:14Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] hostIP: 10.0.133.142 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.133.142 [e2e-llm-inference-service] podIP: 10.132.0.48 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.48 [e2e-llm-inference-service] startTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T18:09:14Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://669fcca57cec4a33eabcb6b6521c37237ac8b5d94e6fb35ad9e15d9436fe768c [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-f66n2 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a175d464-b6e6-41ee-a9d8-c5d400230273 [e2e-llm-inference-service] resourceVersion: '57325' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.72/23"],"mac_address":"0a:58:0a:86:00:48","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.72/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.72\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:48\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] uid: e27ebbb9-af19-450d-8e4e-f2263d5d1e19 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-177 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e27ebbb9-af19-450d-8e4e-f2263d5d1e19"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.72"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-m8hvk [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef:\ [e2e-llm-inference-service] \ decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-m8hvk [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-135-177.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-xjjgk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-bkcw5 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:38Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] hostIP: 10.0.135.177 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.177 [e2e-llm-inference-service] podIP: 10.134.0.72 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.72 [e2e-llm-inference-service] startTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://2c2f1fa1ef8f2bfc1c08ce1ad84b5b3959a777c334e89e0c93e2595ebc0cfdb2 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-m8hvk [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2feee634-b115-4efa-be68-727a44baa4cd [e2e-llm-inference-service] resourceVersion: '56749' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-bkcw5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-bkcw5"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-xjjgk"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-xjjgk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-bkcw5 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-xjjgk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-bkcw5 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e700e6fb-7794-4c6f-afcf-e0bed93c9166 [e2e-llm-inference-service] resourceVersion: '56710' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2x7tv [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2x7tv"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-xjjgk"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-xjjgk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2x7tv [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-xjjgk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2x7tv [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 76c1cf06-77e8-4802-a55a-2ec038902f51 [e2e-llm-inference-service] resourceVersion: '56774' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.171.217 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.171.217 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a62d772b-30fa-4797-8841-38f42e4f469e [e2e-llm-inference-service] resourceVersion: '56731' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.86.158 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.86.158 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ad444a9f-9147-44d8-9ea8-59a41b8a9ee1 [e2e-llm-inference-service] resourceVersion: '63162' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:18:37Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:18:37Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:18:37Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: b61bab4e-7769-4fee-8663-1499e270eff9 [e2e-llm-inference-service] resourceVersion: '58644' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ff9bbb72-3fb7-42c4-a4e4-07a21527af7c [e2e-llm-inference-service] resourceVersion: '57329' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9c651a42-2db4-4a23-8fbb-50ade303c305 [e2e-llm-inference-service] resourceVersion: '56742' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: ad444a9f-9147-44d8-9ea8-59a41b8a9ee1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ad444a9f-9147-44d8-9ea8-59a41b8a9ee1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 580a158a-ef00-44a6-9d08-ecd92b5b19ce [e2e-llm-inference-service] resourceVersion: '58642' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: b61bab4e-7769-4fee-8663-1499e270eff9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b61bab4e-7769-4fee-8663-1499e270eff9"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e27ebbb9-af19-450d-8e4e-f2263d5d1e19 [e2e-llm-inference-service] resourceVersion: '57327' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: ff9bbb72-3fb7-42c4-a4e4-07a21527af7c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ff9bbb72-3fb7-42c4-a4e4-07a21527af7c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9e3d00fc-173f-4443-bc48-e1399d90cbce [e2e-llm-inference-service] resourceVersion: '56768' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 1288054e-1325-442d-8a07-ef8ac930f4ea [e2e-llm-inference-service] resourceVersion: '56717' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 3be8d9e2-584e-41b5-908a-757c5e861a1e [e2e-llm-inference-service] resourceVersion: '56763' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4478e4a4-b571-483f-b9da-f4c47bb419e2 [e2e-llm-inference-service] resourceVersion: '56715' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-hbw2q [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 51469899-87d7-4ad6-bbeb-3c5e2ecc198b [e2e-llm-inference-service] resourceVersion: '57328' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: 76c1cf06-77e8-4802-a55a-2ec038902f51 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:09:09Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"76c1cf06-77e8-4802-a55a-2ec038902f51"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.72 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr [e2e-llm-inference-service] uid: a175d464-b6e6-41ee-a9d8-c5d400230273 [e2e-llm-inference-service] nodeName: ip-10-0-135-177.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-s48295 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 94d3438c-8d70-43e2-be99-39b93ea5519b [e2e-llm-inference-service] resourceVersion: '58640' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: a62d772b-30fa-4797-8841-38f42e4f469e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:11:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a62d772b-30fa-4797-8841-38f42e4f469e"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.48 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f [e2e-llm-inference-service] uid: e40a0463-3c22-4106-a788-24bb2ce11bfc [e2e-llm-inference-service] nodeName: ip-10-0-133-142.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9e3d00fc-173f-4443-bc48-e1399d90cbce [e2e-llm-inference-service] resourceVersion: '56768' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 1288054e-1325-442d-8a07-ef8ac930f4ea [e2e-llm-inference-service] resourceVersion: '56717' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 3be8d9e2-584e-41b5-908a-757c5e861a1e [e2e-llm-inference-service] resourceVersion: '56763' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4478e4a4-b571-483f-b9da-f4c47bb419e2 [e2e-llm-inference-service] resourceVersion: '56715' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '57069' [e2e-llm-inference-service] uid: 61fa8cf3-24b2-4b66-98bb-673d7b2bef76 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '57069' [e2e-llm-inference-service] uid: 61fa8cf3-24b2-4b66-98bb-673d7b2bef76 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '57053' [e2e-llm-inference-service] uid: a73981bd-c505-4a03-baa4-674a303af36d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:49Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T18:08:39Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '56935' [e2e-llm-inference-service] uid: 3386710c-94a1-4174-808b-933b6a31ee96 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:38Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T18:08:39Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '56799' [e2e-llm-inference-service] uid: 10cc55c1-8780-47d9-88f0-f8b09f0750b8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:54Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:54Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '57140' [e2e-llm-inference-service] uid: d28c1a59-4064-4ce5-9476-7b5cdee04dc2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '56848' [e2e-llm-inference-service] uid: 67d3b24e-3924-4738-9996-46f8284ff84c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '56799' [e2e-llm-inference-service] uid: 10cc55c1-8780-47d9-88f0-f8b09f0750b8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:54Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:54Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '57140' [e2e-llm-inference-service] uid: d28c1a59-4064-4ce5-9476-7b5cdee04dc2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '56848' [e2e-llm-inference-service] uid: 67d3b24e-3924-4738-9996-46f8284ff84c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '56799' [e2e-llm-inference-service] uid: 10cc55c1-8780-47d9-88f0-f8b09f0750b8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:54Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:54Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '57140' [e2e-llm-inference-service] uid: d28c1a59-4064-4ce5-9476-7b5cdee04dc2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '56848' [e2e-llm-inference-service] uid: 67d3b24e-3924-4738-9996-46f8284ff84c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"345ed91c-304b-4405-b837-fac607cf5b10"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T18:08:36Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 345ed91c-304b-4405-b837-fac607cf5b10 [e2e-llm-inference-service] resourceVersion: '56786' [e2e-llm-inference-service] uid: 85965996-96ab-43a8-8388-16ee9745a6f0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bnc62f [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:23:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] timestamp: '2026-07-28T18:23:21Z' [e2e-llm-inference-service] window: 19.897s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 98961350n [e2e-llm-inference-service] memory: 2408960Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T18:23:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] timestamp: '2026-07-28T18:23:14Z' [e2e-llm-inference-service] window: 19s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 54754789n [e2e-llm-inference-service] memory: 35004Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-28T18:23:30.607082] end - ❌ 903.114s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:11:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:09:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:08:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-48639af5 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_lws(test_case: TestCase): [e2e-llm-inference-service] """HPA + LWS: HPA exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:670: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...ale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:19:00.853252', start_time = 1785262740.8535752 [e2e-llm-inference-service] duration = 900.9276235103607, timestamp_end = '2026-07-28T18:34:01.781204' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f5ad39d4e00> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_lws] [2026-07-28T18:19:00.487727] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:19:00.500814] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:19:00.853096] end - ✅ in 0.352s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:19:00.853252] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:34:01.781204] end - ❌ 900.928s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:34:01.781330] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-7kdlq'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:34:32.097969] end - ✅ in 30.316s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_lws] [2026-07-28T18:34:32.098170] end - ❌ 931.610s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:20:08Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:36Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:42Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_stop_feature[router-managed-workload-single-cpu-model-fb-opt-125m] __ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name=... {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="stop-feature-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_stop_feature(test_case: TestCase): [e2e-llm-inference-service] """Test that stopping an LLMInferenceService sets the Ready condition to False with reason Stopped.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Create the service [e2e-llm-inference-service] print(f"Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] [e2e-llm-inference-service] # Wait for the service to be ready [e2e-llm-inference-service] print(f"Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:80: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:27:28.144705', start_time = 1785263248.1449947 [e2e-llm-inference-service] duration = 900.8664395809174, timestamp_end = '2026-07-28T18:42:29.011437' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f384cf25120> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-stop-feature-d9eef2a1 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-stop-feature-tes-eeaa3ea3 in namespace e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-stop-feature-tes-eeaa3ea3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-stop-feature-tes-eeaa3ea3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-single-cpu-stop-featur-e7f09208 in namespace e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-stop-featur-e7f09208 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-stop-featur-e7f09208 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-fb-opt-125m-stop-feature-8f213f2f in namespace e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-stop-feature-8f213f2f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-stop-feature-8f213f2f [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_stop_feature] [2026-07-28T18:27:27.877219] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='stop-feature-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-stop-feature-d9eef2a1', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:27:27.890157] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:27:28.144406] end - ✅ in 0.254s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:27:28.144705] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:27:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'message': 'Inference Pool e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:36:18Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 12: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:36:18Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 13: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:36:18Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:42:29.011437] end - ❌ 900.866s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_stop_feature] [2026-07-28T18:42:29.011535] end - ❌ 901.134s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:27:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:28:34Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:28:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-stop-feature-d9eef2a1 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_lws(test_case: TestCase): [e2e-llm-inference-service] """KEDA + LWS: ScaledObject exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:728: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:34:33.544551', start_time = 1785263673.5448844 [e2e-llm-inference-service] duration = 900.8007864952087, timestamp_end = '2026-07-28T18:49:34.345673' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f5ad39d5800> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_lws] [2026-07-28T18:34:33.083549] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:34:33.097499] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:34:33.544407] end - ✅ in 0.447s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:34:33.544551] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:34:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'message': 'ScaledObject conditions not yet available', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:49:34.345673] end - ❌ 900.801s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:49:34.345782] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-cv72w'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:49:59.958703] end - ✅ in 25.612s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_lws] [2026-07-28T18:49:59.958847] end - ❌ 926.875s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:35:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:35:03Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-lws-e541a132 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_cleanup_hpa[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-cleanup-hpa", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_cleanup_hpa(test_case: TestCase): [e2e-llm-inference-service] """Removing scaling config should delete HPA.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:892: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:50:01.350551', start_time = 1785264601.350893 [e2e-llm-inference-service] duration = 900.9048171043396, timestamp_end = '2026-07-28T19:05:02.255719' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f5ad39d6e80> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_cleanup_hpa] [2026-07-28T18:50:00.982403] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-cleanup-hpa', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:50:00.995865] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:50:01.350404] end - ✅ in 0.354s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:50:01.350551] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:50:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T19:05:02.255719] end - ❌ 900.905s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T19:05:02.255881] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-qjlr5'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T19:05:32.577488] end - ✅ in 30.321s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_cleanup_hpa] [2026-07-28T19:05:32.577672] end - ❌ 931.595s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:50:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:51:27Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:51:09Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:92: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 7 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 7 failed, 42 passed, 3 skipped, 25 warnings in 5390.95s (1:29:50) ======= [must-gather] [must-gather ] OUT 2026-07-28T19:05:34.670990765Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 3ff16daa-3fce-4b00-be0a-650b5c215b4a [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T19:05:34.721498556Z namespace/openshift-must-gather-jhlmk created [must-gather] [must-gather ] OUT 2026-07-28T19:05:34.729196643Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-88kls created [must-gather] [must-gather ] OUT 2026-07-28T19:05:34.756676778Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:41.460869782Z [disk usage checker] Started [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:41.464318493Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:41.740799424Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:41.909351830Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:41.909383349Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:41.909383349Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:41.912162036Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.087304830Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.087333762Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.087333762Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.090212754Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.255923344Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.255950939Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.255950939Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.259097852Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.431735956Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.431767329Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.431767329Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.435007024Z Error getting logs from rhods-notebooks [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.599575262Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.599680528Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.599680528Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.603464105Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.777749052Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.777850915Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.777850915Z namespaces "istio-system" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.780611943Z Error getting logs from istio-system [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.951078303Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.951108761Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.951108761Z namespaces "knative-serving" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:42.953859776Z Error getting logs from knative-serving [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.124057019Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.124087184Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.124087184Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.127091875Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.423010772Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.719882604Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.864567136Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.950093112Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:43.952616119Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.089025253Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.173148674Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.175499238Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.309028384Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.392503860Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.394975994Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.534248092Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.617295794Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.619821946Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.757866794Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.843169732Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.845677194Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:44.987874739Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.076315742Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.079157427Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.215395600Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.298820637Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.301430116Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.444281408Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.527051897Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.529841741Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.667251549Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.750193713Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.752775122Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.896383016Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.980702572Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:45.983334968Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.123720490Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.205668022Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.208295866Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.342384171Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.423965879Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.426541861Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.468762179Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.564695843Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.647337659Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.649853349Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.788339035Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.870625637Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:46.873319155Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.009196012Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.091091137Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.093625456Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.231729220Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.314360473Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.317060128Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.453431455Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.540831865Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.543853792Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.679359489Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.762325159Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:47.764903694Z Error collecting info from [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:48.492981133Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:48.626429847Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:48.761928497Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:48.903951842Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:49.037641821Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:49.641309635Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:49.777001470Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:49.920171049Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:50.065901625Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:50.199127150Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:50.764762226Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:50.912480755Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.010858121Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.088637031Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.263424851Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.417928570Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.474332924Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.520076032Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.604070447Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.775938489Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:51.926188897Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:52.023418581Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:52.102547578Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:52.271229006Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:52.423029280Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:52.517018887Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:52.598335624Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:52.764860912Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:52.912713761Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:53.041648097Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:53.118939288Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:53.297783364Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:53.448484694Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:53.540395650Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:53.624490187Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:53.790038155Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:53.929817066Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:54.024179082Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:54.113799459Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:54.298704596Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:54.451193951Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:54.544405934Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:54.630070409Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:54.795676085Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:54.933071308Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:55.021163890Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:55.093330934Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:55.260729014Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:55.416190528Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:55.509493937Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:55.585655528Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:55.759000075Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:55.906491572Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.000611196Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.076844213Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.246037641Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.388054538Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.479547779Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.481448881Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.561413301Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.728739131Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.873921339Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:56.963002647Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:57.037302763Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:57.205582690Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:57.359977463Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:57.451519477Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:57.527583626Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:57.697573815Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:58.376053614Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:58.540017741Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:58.645325208Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:58.814078104Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:59.483601939Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:59.643223338Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:59.751639657Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:05:59.918458479Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:00.600924983Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:00.757667328Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:00.881697916Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:01.055090847Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:01.484335674Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:01.757682077Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:01.930992764Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:02.043204055Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:02.213075414Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:02.959605689Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:03.178969315Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:03.301905625Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:03.475234929Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:04.406938979Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:04.619417282Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:04.758867477Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:04.925581833Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:05.630787172Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:05.821870495Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:05.943074875Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:06.106324648Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:06.488871436Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:06.821668968Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:07.009024430Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:07.130560935Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:07.296380635Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:08.013911187Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:08.204546613Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:08.326235272Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:08.491291127Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:09.200472419Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:09.398557346Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:09.524104307Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:09.697434349Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:10.404535134Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:10.589020643Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:10.707072959Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:10.876173663Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:11.496597340Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:11.582336437Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:11.772524159Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:11.892758373Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:12.072272875Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:12.784975342Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:12.975969059Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:13.107177333Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:13.283795951Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:14.002853937Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:14.201839467Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:14.328445556Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:14.493024724Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:15.223702519Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:15.426503310Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:15.550926441Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:15.724064070Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:16.428310196Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:16.503068571Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:16.615285206Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:16.737397043Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:16.905778962Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:17.548414817Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:17.737194277Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:17.861930319Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:18.028850362Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:18.760348023Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:18.952829656Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:19.076723272Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:19.249785425Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:19.956761409Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:20.144421790Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:20.265251864Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:20.435168668Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:21.241004374Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:21.489447497Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:21.508887357Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:21.609991881Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:21.778000147Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:22.483703274Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:22.673695103Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:22.795113161Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:22.963592906Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:23.660434736Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:23.849418825Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:23.970701323Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:24.141393339Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:24.845167557Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:25.049829589Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:25.170394047Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:25.339219480Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:26.045199928Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:26.234030759Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:26.354464884Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:26.513373862Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:26.522909370Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:27.234940463Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:27.426343948Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:27.552576291Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:27.717599345Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:28.437167919Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:28.674370400Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:28.800204507Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:28.969933511Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:29.681510609Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:29.869068022Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:29.987463355Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:30.166006592Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:30.912746765Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:31.111734680Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:31.283379320Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:31.456997693Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:31.519170835Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:32.220917471Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:32.432116691Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:32.553265322Z Wrote inspect data to must-gather. [must-gather] [must-gather-2lkr5] POD 2026-07-28T19:06:32.577099276Z Caches written to disk [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.586336171Z waiting for gather to complete [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.589942633Z downloading gather output [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.879568047Z receiving incremental file list [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.89482177Z ./ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.894908966Z aggregated-discovery-api.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.895198625Z aggregated-discovery-apis.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.896345362Z event-filter.html [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.896720968Z timestamp [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.896869038Z version [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900384973Z cluster-scoped-resources/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900403325Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900413135Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900437197Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900505001Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900521173Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900532454Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900600478Z namespaces/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900612229Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900656832Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900728007Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900766239Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900811592Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.900852275Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.90093362Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901017686Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901095801Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901134924Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901179957Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901215499Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901270243Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901346228Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901397292Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901417553Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901513129Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901550332Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901588414Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901740584Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901804579Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.901878564Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.90196805Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902042425Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902160693Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902346935Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902393538Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902437041Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902481744Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902523637Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.90256644Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902610703Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.90271508Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902763513Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902805106Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902852429Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902886581Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902937475Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.902980498Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.903026481Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.903045902Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.903116417Z namespaces/kuadrant-system/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.903146399Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.903198872Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.903244135Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.903291648Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.90332111Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.903408106Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.90375676Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904080772Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904120384Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904164777Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.9042112Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904249633Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904333649Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904382222Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904417494Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.9045031Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904553813Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904604827Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904794669Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.904917558Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.905561861Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.905657207Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906035613Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906159891Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906385266Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906515355Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906541677Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906692877Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906719359Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906781663Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906812555Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.90689777Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906926832Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.906978426Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.90703922Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.907065042Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.907106875Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.907117205Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.907130346Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino-7cf4765b85-5shw9.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.907232683Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.907242444Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.907247764Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.907283186Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.920663154Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.920732749Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.92074436Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.920805284Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/authorino-operator-56fb6f6856-r8p5c.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.920906721Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.920916951Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.920921762Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.920944313Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921155097Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921221572Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921234223Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921291236Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/dns-operator-controller-manager-ccbdf686-nkbjm.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921387953Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921399304Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921404674Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921430836Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921537233Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921602727Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921625669Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921712645Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin-598f86468b-99wjj.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.92179657Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921806761Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921811691Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921841943Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921921599Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.921984603Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.922029326Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.922048497Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/kuadrant-operator-controller-manager-66866f5fcc-4zxvt.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.922205348Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.922214928Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.922219319Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.92224058Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934040182Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934113657Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.93415542Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934230935Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador-limitador-85d9db4d5d-zllks.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934335752Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934345222Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934350613Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934403266Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934507013Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934584168Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934598069Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934694756Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/limitador-operator-controller-manager-774d94476f-dzzbw.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934803143Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934812254Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934817294Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.934863867Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935083722Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935170668Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.93520916Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935269454Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935318248Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.93535468Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935408504Z namespaces/openshift-ingress/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935455947Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935522831Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935556424Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935607967Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935665931Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.935757937Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936005214Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936200347Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936248Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936283272Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936346057Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936375739Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936461934Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936511608Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.93655123Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936628325Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936696Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.936718542Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.937352854Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.937485483Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.937720259Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.937798844Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938054371Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.93819213Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938413605Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.93848612Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938505611Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938602048Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938623819Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938696154Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938730837Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938813912Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938868406Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938911219Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938953641Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.938998744Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.939049868Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.939066639Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.93908641Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/istiod-openshift-gateway-5677ccb49b-82lr6.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.939197038Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.939206309Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.939210839Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.939239471Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.954559449Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.954655215Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.954680037Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.954770033Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.954896411Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.954907102Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.954912292Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.954935894Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.95607855Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956165907Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956197309Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router-default-5657cc6765-srj79.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956311036Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956318907Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956323057Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956345688Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/current.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956470927Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/previous.insecure.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956537021Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/previous.log [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956560283Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956609236Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.95665944Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.956711913Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.966108354Z [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.966126315Z sent 2,836 bytes received 977,294 bytes 1,960,260.00 bytes/sec [must-gather] [must-gather-2lkr5] OUT 2026-07-28T19:06:33.966140496Z total size is 14,328,988 speedup is 14.62 [must-gather] [must-gather ] OUT 2026-07-28T19:06:34.020363684Z namespace/openshift-must-gather-jhlmk deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 3ff16daa-3fce-4b00-be0a-650b5c215b4a [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather ] OUT 2026-07-28T19:06:34.154633544Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 3ff16daa-3fce-4b00-be0a-650b5c215b4a [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T19:06:34.177285484Z namespace/openshift-must-gather-rd9v9 created [must-gather] [must-gather ] OUT 2026-07-28T19:06:34.182134439Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-hshvx created [must-gather] [must-gather ] OUT 2026-07-28T19:06:34.267996811Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:35.603722588Z [disk usage checker] Started [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:35.607027261Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:35.826876273Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:35.992795896Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:36.110067993Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:36.212395141Z Gathering data for ns/default... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:36.594786194Z Gathering data for ns/openshift... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:37.099444833Z Gathering data for ns/kube-system... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.076938066Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.477301152Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.477340910Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.477340910Z namespaces "assisted-installer" not found [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.716217348Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.725975277Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.728155807Z INFO: Gathering HAProxy config files [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.728586757Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.732158633Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.748315215Z Executing Istio gather script [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.763872425Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.766598106Z INFO: Collecting host service logs for crio [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.766935151Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.767256013Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.767569682Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.767898441Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.768261319Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.768641183Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.769207310Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.769541309Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.769963008Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.770304249Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:38.770669508Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.296282555Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.466111959Z No resources found [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.538659241Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-msvxc [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.538705174Z insights-runtime-extractor-czx2d [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.538727795Z insights-runtime-extractor-nvtkq [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.538748446Z insights-runtime-extractor-shhb8 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.737283408Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.752500234Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.869903037Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:39.968648837Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:40.198913169Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:40.414828144Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:40.452124544Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:40.452231823Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:40.654200106Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:40.756652361Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:40.789658815Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:40.997272663Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.039672339Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.039672339Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.326529109Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.385884777Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.403342199Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.434649988Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.532162879Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.643339377Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.645009141Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.678057889Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.962226128Z No resources found [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.994180503Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:41.998725151Z tar: Removing leading `/' from member names [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.005494918Z INFO: HAProxy config collection complete. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.171711847Z INFO: 'previous-logs' folder not found on ip-10-0-131-29.ec2.internal, skipping... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.277617307Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.352404234Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.414449193Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.415858332Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.750322824Z INFO: 'previous-logs' folder not found on ip-10-0-133-142.ec2.internal, skipping... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.756043837Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.786785607Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.934606288Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.990901482Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.993327372Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:42.996931214Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.292072608Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.367767815Z INFO: 'previous-logs' folder not found on ip-10-0-135-177.ec2.internal, skipping... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.367896413Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.367961870Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.510949198Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.514384221Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.519094356Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.633991028Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.723713742Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.866247189Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:43.880379020Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.081629207Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.082066641Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.087899434Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.316160295Z INFO: INTERCONNECT MODE [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.316307899Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.365573140Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.432687412Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.436883907Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.567346970Z INFO: Gathering OVN_Northbound from ovnkube-node-64lk7... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.567861746Z INFO: Gathering OVN_Northbound from ovnkube-node-b648k... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.568197037Z INFO: Gathering OVN_Northbound from ovnkube-node-nrwj7... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.568610325Z INFO: Gathering OVN_Southbound from ovnkube-node-64lk7... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.568932972Z INFO: Gathering OVN_Southbound from ovnkube-node-b648k... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.569206842Z INFO: Gathering OVN_Southbound from ovnkube-node-nrwj7... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.575035264Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.798371704Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.918918982Z tar: Removing leading `/' from member names [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.922241019Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.929303635Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.952651474Z tar: Removing leading `/' from member names [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:44.962543795Z tar: Removing leading `/' from member names [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.021299584Z tar: Removing leading `/' from member names [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.057200116Z tar: Removing leading `/' from member names [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.073659176Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.091888129Z tar: Removing leading `/' from member names [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.293430379Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.304812135Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.404181089Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.678139989Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.756337254Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.756383861Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.756979763Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.758840443Z 7.0M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.761082841Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.766371214Z ovnk_database_store/ [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.766474402Z ovnk_database_store/ovnkube-node-64lk7_nbdb [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.786099154Z ovnk_database_store/ovnkube-node-nrwj7_nbdb [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.807949481Z ovnk_database_store/ovnkube-node-64lk7_sbdb [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.812834440Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.871626725Z ovnk_database_store/ovnkube-node-b648k_sbdb [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.928700672Z ovnk_database_store/ovnkube-node-nrwj7_sbdb [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.974039477Z ovnk_database_store/ovnkube-node-b648k_nbdb [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:45.990540855Z INFO: Network log collection complete. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:46.068206012Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:46.288082404Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:46.511447404Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:46.639032492Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:46.715545683Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:46.888936976Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:47.210818748Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:47.213263435Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:47.224490078Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:47.687004553Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:47.754276429Z Gathering data for ns/openshift-console... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:47.925274452Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.017799115Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.158933089Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.162183646Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.181300016Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.358866831Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.480193127Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.481679162Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.768437161Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.772673949Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:48.997474399Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.151653223Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.268450212Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.270961991Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.485189186Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.531815342Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.551908148Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.708669494Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.711666447Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:49.929329179Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.164296479Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.168200335Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.301080999Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.382164129Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.617252473Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.621369424Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.678338153Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.817866073Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.841072502Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:50.968184853Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:51.091921046Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:51.094251029Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:51.344547205Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:51.423160133Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:51.579807077Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:51.582565009Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:51.777612258Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:51.824654580Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.002249856Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.006722995Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.215885780Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.463853957Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.470306004Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.473588370Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.671068511Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.912064284Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.914499565Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:52.980578459Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:53.006488606Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:53.124388079Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:53.357442685Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:53.360350111Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:53.551361649Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:53.796614300Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:53.799302393Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:53.980422564Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:54.140209712Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:54.204341873Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:54.206497804Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:54.401216248Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:54.618785838Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:54.621846788Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:54.818472740Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.049503620Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.052382347Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.268149920Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.306932285Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.493536295Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.496395101Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.713887409Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.823103460Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.945136283Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:55.947608378Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.149899595Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.264958959Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.381631675Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.385490237Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.482945848Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.633654119Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.811915433Z Collecting performance related data for node ip-10-0-133-142.ec2.internal [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.821004609Z Collecting performance related data for node ip-10-0-131-29.ec2.internal [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.833085318Z Collecting performance related data for node ip-10-0-135-177.ec2.internal [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.929788409Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:56.934400794Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:57.039827314Z Gathering data for ns/kserve... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:57.178236199Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:57.453641430Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:57.600676744Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:57.860317201Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:58.789898953Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:59.008705562Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:59.173170428Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:59.323983755Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:06:59.576563518Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:00.309969212Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:00.458119741Z Gathering data for ns/openshift-config... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:00.561703205Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:00.717508494Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:00.827863587Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:00.896037486Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:01.267773508Z Done executing Istio gather script [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:01.427916991Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:01.774973957Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:02.124037223Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:02.519163339Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:02.914001774Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:03.274132909Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:03.647498902Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:04.089700486Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:04.464639208Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:05.117752120Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:05.833086861Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:06.356152707Z Collecting kubelet logs for node ip-10-0-133-142.ec2.internal [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:06.357938008Z Collecting kubelet logs for node ip-10-0-131-29.ec2.internal [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:06.359596874Z Collecting kubelet logs for node ip-10-0-135-177.ec2.internal [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:06.839368683Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-rd9v9 namespace [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:06.845148529Z INFO: Node performance data collection complete. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:06.964355759Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:08.816009886Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:09.183080024Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:09.823789424Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:10.296604814Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:10.722273763Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:10.837886276Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:11.072869981Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:11.624746099Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:11.974225010Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:12.460628494Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:12.795981963Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:13.138910101Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:13.879835215Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:14.252166289Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:14.678704795Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:15.132558916Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:15.842889433Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:16.225830614Z Wrote inspect data to must-gather. [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:16.225878878Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:16.225878878Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:16.374185047Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-tn7p8] POD 2026-07-28T19:07:16.547226257Z Caches written to disk [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.559892642Z waiting for gather to complete [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.564709225Z downloading gather output [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.857222437Z receiving incremental file list [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.879745986Z ./ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.879821021Z aggregated-discovery-api.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.880170225Z aggregated-discovery-apis.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.881102347Z event-filter.html [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.88397029Z timestamp [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.884064056Z version [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.891993897Z cluster-scoped-resources/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.89204652Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892067502Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892169139Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892401404Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892501361Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892565005Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892612669Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892730646Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892810562Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892923289Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.892967832Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.893105002Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.893177776Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.893635037Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.893698061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.893902385Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.894167503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.894358686Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.894499975Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.898341173Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.900143133Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.900257701Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.900380329Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.900481866Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.900610565Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.90114288Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.901304591Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.901464382Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.901541617Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.902359172Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.902593197Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.902742787Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.902911189Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.904088148Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.904311213Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.904474224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.904629754Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.910619735Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.910801217Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.911346364Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.911662285Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.912515542Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.912666752Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.912815552Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.91293299Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.913047178Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.913155105Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.913330437Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.91353358Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.914443752Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.914603482Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.914777164Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.916934068Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.917134372Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.917300733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.917570101Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.918060114Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.918510174Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.918658584Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.918802904Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.918927162Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.91905348Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.91919927Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.919330229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.919506231Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.919670752Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.919835583Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.920104441Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.920266462Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.92038779Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.920954988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.92114158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.921269809Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.921462182Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.922232803Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.922360012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.922556045Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.922714846Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.922882237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.923012876Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.923171976Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.923314816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.923445885Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.923580614Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.923733534Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.923871733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.924022593Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.924193115Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.924356316Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.924507056Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.924661736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.925068904Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.925246265Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.925700286Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.925839585Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.926905997Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.927042586Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.927158133Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.927262671Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.927382139Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.927583222Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.927873721Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.927991889Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.928127129Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.928271448Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.928381895Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.928505354Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.928673395Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.928805184Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.932812943Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.933274073Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.93396586Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.934155562Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.934317283Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.934448842Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.934582511Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.934746932Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.934848359Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.935622451Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.93605266Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.938997727Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.939130176Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.939292247Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.939442037Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.939580126Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.939713665Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.939847804Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.942560616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.945712967Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.953141985Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.960971459Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.96112792Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.961515746Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.961802195Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.961951875Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.962119676Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.962282047Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.962559536Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.962710726Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.962862666Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.96307514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.96321788Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.963337208Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.963453466Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.963627608Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.963799849Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.963951979Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.964070747Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.964180745Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.964310443Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.964512537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.964673907Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.96531224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.965437889Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.965569607Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.965895479Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.966141616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.966830062Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.966986953Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.967271262Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.96754691Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.970878783Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.971107129Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.971235207Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.971376067Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.971523866Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.971954375Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.972157619Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.972393455Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.972541225Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.975563027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.97589931Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.976183099Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.976397653Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.976569515Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.97695364Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.978371896Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.979351861Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.97993112Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.980194267Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.980502968Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.9806794Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.981564149Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.981803275Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.981933574Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.9836608Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.983859383Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.984315754Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.984462904Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.98470682Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.985120258Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.985253887Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.985538156Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.986035029Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.986167678Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.986324778Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.986518741Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.986730176Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.986898157Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.987084319Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.987256411Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.987677599Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.987754064Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.987854661Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.987974719Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.988085636Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.988199244Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.98843951Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.988564719Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.988702608Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.988809625Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.988896041Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989021799Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989118536Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989197121Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989286837Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989377603Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.98947643Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989556945Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989680273Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.98978143Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989866716Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.989955772Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.990042958Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.990134794Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.990219069Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.990310425Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.990399902Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.990810269Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.990893524Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.990982141Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991067096Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991154982Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991245348Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991352525Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991438371Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991534507Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991622223Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991819266Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991910853Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.991999219Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.992084864Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.99217276Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.992258516Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.992345632Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.992448939Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.992532534Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.992628451Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.99277123Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.992898929Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993020027Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993100203Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993197439Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993295096Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993384722Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993486948Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993586875Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993725614Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993876295Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.993972261Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.99411241Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.994244859Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.994355427Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.994464154Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.994564931Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.994686389Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.994812117Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.994906513Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.99500867Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.99514795Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.995251817Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.995345153Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.995432899Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.995534936Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.995614111Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.995763021Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.995847027Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.995944123Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.99604822Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.996137196Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.996223732Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.996326799Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.996419855Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.996537423Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.996676122Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.996822262Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.996935429Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.997061948Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.997193467Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.997350687Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.997458955Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.997565792Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.997682379Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.997806908Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.997911885Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998012842Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998102628Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998187543Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998302671Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998406318Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998490204Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998592751Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998715629Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998818756Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.998913572Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999014759Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999117226Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999205882Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999381003Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999422316Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999592458Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999769039Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999874286Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:17.999960772Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.000293374Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.000410332Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.000497648Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.000593034Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.000738824Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00081888Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.000910786Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001022413Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001109239Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001193945Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001290481Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001388998Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001486094Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00157541Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001675027Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001784724Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00186954Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.001957956Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002061443Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002110916Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002213763Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00230772Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002406126Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002490762Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002582898Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002717677Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002874747Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.002964224Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00305789Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.003155246Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.003286975Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.003422534Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00351405Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.003610827Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.003737975Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.003831462Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.003924978Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004014034Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00410127Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004197096Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00425461Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004306374Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004366867Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00440642Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004488486Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00454827Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004728702Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004767154Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.0048483Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004888053Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004937836Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.004994409Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005043503Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005059964Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005112328Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005204544Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005289709Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005357254Z cluster-scoped-resources/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005365534Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005416808Z cluster-scoped-resources/core/nodes/ip-10-0-131-29.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00560303Z cluster-scoped-resources/core/nodes/ip-10-0-133-142.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005784753Z cluster-scoped-resources/core/nodes/ip-10-0-135-177.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005880129Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005929662Z cluster-scoped-resources/core/persistentvolumes/pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.005985726Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006071972Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006128225Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006216081Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006297567Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006378632Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006466758Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006548864Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006739746Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006840683Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.006988263Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007075219Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007169245Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007260001Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007354127Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007438023Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00753352Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007617345Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007745344Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007823799Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.007917845Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008009282Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008093597Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008146521Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008187073Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008274609Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008370716Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008453191Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008537647Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008619522Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008816465Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.008954775Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009112956Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009138067Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009148758Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00917676Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009250685Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009264926Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009369013Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009498871Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009518893Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009608809Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.00962229Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009703075Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009810472Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.009967313Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010038827Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010053988Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010103322Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010213849Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.0102214Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010265613Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010360459Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010403492Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010537521Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010582144Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.0108156Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.010886364Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01096422Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011024273Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011148892Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011193625Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011248159Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011290161Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011342095Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011382577Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011434811Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011472504Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01157445Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011614563Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011734631Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011774974Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011843248Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011880971Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011944125Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.011990568Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012050352Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012088815Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012193012Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012207233Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012243695Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012326031Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012369734Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012507193Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01262026Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012819254Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.012922561Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01306178Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013175018Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013270574Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013341279Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013390142Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013479348Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013565824Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013752466Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013898556Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.013994112Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.014081048Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.014235909Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.014325225Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.014419251Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.014507097Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.014600743Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.014749813Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01484502Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.014939696Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.015023632Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.015114028Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.015209944Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01529293Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.015379556Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.015496593Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.0156031Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.015678576Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.015821715Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.015906551Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016001167Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016151347Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016256944Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016351361Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016443167Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016530133Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016625659Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016749777Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.016852874Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01693915Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017040387Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017131633Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017219249Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017309475Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017396811Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017490027Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017585313Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017707241Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017815379Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.017898094Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018017422Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018112268Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018207205Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018299181Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018398248Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018483493Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01858164Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018659825Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018678566Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.01872968Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018804175Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018811445Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018855378Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018924293Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018934294Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.018971196Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019029Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019062592Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-29.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019164549Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-133-142.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019242404Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-135-177.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019300978Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019340551Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019429267Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019488791Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019528203Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-79bfcbce2c303a7be4e0da631f9325c7283bc1a4d6c0ccaf8fcae15a467ca4d1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019599508Z host_service_logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019635081Z host_service_logs/masters/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019721847Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019798922Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.019861706Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.0199264Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020063989Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020169826Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020244402Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020314046Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020387181Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020455926Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020528321Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020573773Z ingress_controllers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020588974Z ingress_controllers/default/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020596705Z ingress_controllers/default/router-default-5657cc6765-srj79/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020625047Z ingress_controllers/default/router-default-5657cc6765-srj79/haproxy.config [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020793288Z insights-data/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020832911Z istio/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.020881074Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.021014623Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.02156327Z istio/event-filter.html [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.021759513Z istio/timestamp [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.02186808Z istio/cluster-scoped-resources/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.021878431Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.021882531Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.021920754Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.021990139Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.022041902Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.022109266Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.02217058Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.022216984Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.022386415Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.02350157Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.023789439Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.023967501Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.02440349Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.024579662Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.025042813Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.026168619Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.026316438Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.026443767Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.026577816Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.026724856Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.026845824Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.026999664Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.027185276Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.027452134Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.027772486Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028255188Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028397698Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028531777Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028711679Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.02872275Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028757532Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028818406Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028827017Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028856688Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.028950685Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029135127Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029210872Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029258855Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029366393Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029578037Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.02963288Z istio/namespaces/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029673703Z istio/namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029693905Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029703405Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029767209Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029852325Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029862006Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029899418Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029959422Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.029966273Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030002455Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.03007184Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030083881Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030123144Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.03022151Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030378121Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030444865Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030456656Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/gateway.networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030462446Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030504459Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/gateway.networking.k8s.io/httproutes/stop-feature-test-kserve-route.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030590555Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030598385Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030633188Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.k8s.io/inferencepools/stop-feature-test-inference-pool.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030743315Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.x-k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030759686Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030799309Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.x-k8s.io/inferencepools/stop-feature-test-inference-pool.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030867443Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030874094Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/destinationrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.030916997Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/destinationrules/stop-feature-test-kserve-scheduler.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.031005123Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/destinationrules/stop-feature-test-kserve-shadow-svc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.031087998Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/destinationrules/stop-feature-test-kserve-workload-svc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.03126867Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.031371457Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.031431251Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.031472374Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.031553379Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.031597832Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.031810617Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032070724Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032273348Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.03231472Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032360373Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032435839Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032474221Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032559047Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.03260634Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032659123Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032772361Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.032860957Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.03291098Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.033548003Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.033667471Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.03394538Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034018405Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034277332Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034401Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034604424Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034732562Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034780096Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034861781Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034871592Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.034917285Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035005351Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035050534Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035115098Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035152681Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035241536Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035317352Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035377185Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035418178Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035499894Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035519295Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035562508Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035678346Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035853808Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.035964435Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.036004138Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.036072152Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.036085403Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.036133586Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/istiod-openshift-gateway-5677ccb49b-82lr6.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.036256704Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.036265085Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.036274756Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.036319829Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.045577049Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.045666945Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.045708138Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.045765292Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.04588224Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.04589245Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.04589727Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.045960245Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.046983333Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047071989Z istio/namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047120743Z istio/namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router-default-5657cc6765-srj79.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047216579Z istio/namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047226139Z istio/namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.04723228Z istio/namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047267322Z istio/namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047374949Z istio/namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047441424Z istio/namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047481087Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047507969Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047562802Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047601505Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047658959Z machine_config_ondisk/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047672129Z machine_config_termination_logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.04767635Z monitoring/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.04768021Z monitoring/alertmanager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047725143Z monitoring/alertmanager/status.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.04783091Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047884864Z monitoring/prometheus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.047927846Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.048015603Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.048080547Z monitoring/prometheus/rules.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.048902342Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.048936794Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.048999778Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.04991585Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.049968133Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.050007216Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.050096392Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.050164907Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.050255713Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.050314886Z monitoring/prometheus/status/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.050351059Z monitoring/prometheus/status/config.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.050844582Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.050910897Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051001063Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051225687Z namespaces/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051233758Z namespaces/cert-manager-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051240079Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051243999Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051289112Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051364877Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051372737Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051417451Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051630145Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.051887492Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.052034052Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.052251376Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.052488152Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.052527155Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-lpdf8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.052767771Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.052824055Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.052866397Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.05290235Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.052965294Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053006547Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053089792Z namespaces/cert-manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053098023Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053109594Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053163317Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053309807Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053489439Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053630469Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.053886986Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054117931Z namespaces/default/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054172895Z namespaces/default/default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.05424568Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054283852Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054357177Z namespaces/default/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.05439288Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054483276Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054552781Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054667478Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054724492Z namespaces/default/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054768245Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054828889Z namespaces/default/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054864071Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.054943877Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.05499214Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055037473Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055113198Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055161211Z namespaces/default/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055201504Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055305931Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055389526Z namespaces/default/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055599451Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055774872Z namespaces/default/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055854238Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.055978666Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056149328Z namespaces/default/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056221462Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056259115Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056325209Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056368312Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056411685Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056441727Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056528033Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056602458Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056669002Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056728916Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056798491Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056846214Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056904688Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056914119Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.056960172Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.057145574Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.057333247Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.057469706Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.057701072Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.057896575Z namespaces/default/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.057932117Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.057993001Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058037314Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058094698Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058103468Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058113559Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058160882Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058314262Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058493014Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058628264Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.058880491Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.059069133Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.059076584Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.059088494Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.059127387Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.059278887Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.059459059Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.059592178Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.059818713Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.060012706Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.060020117Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.060027997Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.060080761Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.060235221Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.060419764Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.060553403Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.060772707Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06095715Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06096591Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.061010453Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.061071287Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.061240529Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.061425611Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06155771Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.061792356Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.061973348Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.061982968Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.061993989Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.062041843Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.062195323Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.062381775Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.062514054Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06274943Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.062950133Z namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.062958974Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.062963994Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063009077Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063105674Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063165238Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063216951Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063299477Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06335628Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063366091Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063426975Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063576455Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063775379Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.063912588Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064114481Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064298854Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064306424Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064310384Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064343597Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064429892Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064469355Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064514758Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064598334Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064661818Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064679379Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064739293Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.064879513Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.065090867Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.065242207Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.065449281Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.065634403Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.065658755Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.065701528Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06573401Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.065897471Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.066235203Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.066371933Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.066583227Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.066848785Z namespaces/kserve/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.066893888Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.066971953Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067021536Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06708695Z namespaces/kserve/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067122873Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067208939Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067407132Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067595315Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067658579Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06768417Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067745255Z namespaces/kserve/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067801028Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.067879344Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06797389Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068026264Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068098068Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068143901Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068151492Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068186834Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.06827676Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068363726Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068458362Z namespaces/kserve/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068489584Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068819377Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.068926944Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.069252575Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.069325711Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.069587618Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.069744519Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070013497Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070108853Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070187728Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070282335Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.07036514Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070433725Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070478458Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070560623Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070635408Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070711393Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070736375Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070822511Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070865003Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070954449Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.070997243Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.071038915Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.071102039Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.07111886Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.071167674Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.071321274Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.071495056Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.071629765Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.071895463Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072125388Z namespaces/kserve/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072132608Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-tftvf/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072181632Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-tftvf/kserve-controller-manager-cd4d44c67-tftvf.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072278088Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-tftvf/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072285409Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-tftvf/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072293449Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-tftvf/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072342263Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-tftvf/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.07245766Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-tftvf/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072531705Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-tftvf/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.07259275Z namespaces/kserve/pods/llmisvc-controller-manager-779d8dc47c-bkr24/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072630642Z namespaces/kserve/pods/llmisvc-controller-manager-779d8dc47c-bkr24/llmisvc-controller-manager-779d8dc47c-bkr24.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.07275581Z namespaces/kserve/pods/llmisvc-controller-manager-779d8dc47c-bkr24/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072768621Z namespaces/kserve/pods/llmisvc-controller-manager-779d8dc47c-bkr24/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072773462Z namespaces/kserve/pods/llmisvc-controller-manager-779d8dc47c-bkr24/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.072798123Z namespaces/kserve/pods/llmisvc-controller-manager-779d8dc47c-bkr24/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.079853986Z namespaces/kserve/pods/llmisvc-controller-manager-779d8dc47c-bkr24/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.07991903Z namespaces/kserve/pods/llmisvc-controller-manager-779d8dc47c-bkr24/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.079964753Z namespaces/kserve/pods/model-serving-api-7b954ffc95-hwv4w/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080013127Z namespaces/kserve/pods/model-serving-api-7b954ffc95-hwv4w/model-serving-api-7b954ffc95-hwv4w.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080131215Z namespaces/kserve/pods/model-serving-api-7b954ffc95-hwv4w/server/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080142135Z namespaces/kserve/pods/model-serving-api-7b954ffc95-hwv4w/server/server/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080158626Z namespaces/kserve/pods/model-serving-api-7b954ffc95-hwv4w/server/server/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.08021003Z namespaces/kserve/pods/model-serving-api-7b954ffc95-hwv4w/server/server/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080290955Z namespaces/kserve/pods/model-serving-api-7b954ffc95-hwv4w/server/server/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080385842Z namespaces/kserve/pods/model-serving-api-7b954ffc95-hwv4w/server/server/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080449346Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t2bz8/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.0805135Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t2bz8/odh-model-controller-75d9cdbddf-t2bz8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080605396Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t2bz8/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080621737Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t2bz8/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080635928Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t2bz8/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.080696133Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t2bz8/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.111940796Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t2bz8/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112026812Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t2bz8/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112075325Z namespaces/kserve/pods/s3-init-jmftq/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112145Z namespaces/kserve/pods/s3-init-jmftq/s3-init-jmftq.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112310101Z namespaces/kserve/pods/s3-init-jmftq/download-hf-model/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112321892Z namespaces/kserve/pods/s3-init-jmftq/download-hf-model/download-hf-model/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112328452Z namespaces/kserve/pods/s3-init-jmftq/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112375415Z namespaces/kserve/pods/s3-init-jmftq/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112465941Z namespaces/kserve/pods/s3-init-jmftq/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112536606Z namespaces/kserve/pods/s3-init-jmftq/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112568678Z namespaces/kserve/pods/s3-init-jmftq/s3-init/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112574429Z namespaces/kserve/pods/s3-init-jmftq/s3-init/s3-init/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112579099Z namespaces/kserve/pods/s3-init-jmftq/s3-init/s3-init/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.112655884Z namespaces/kserve/pods/s3-init-jmftq/s3-init/s3-init/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113019159Z namespaces/kserve/pods/s3-init-jmftq/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113231703Z namespaces/kserve/pods/s3-init-jmftq/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113259704Z namespaces/kserve/pods/seaweedfs-64568bcd49-db8st/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.11333743Z namespaces/kserve/pods/seaweedfs-64568bcd49-db8st/seaweedfs-64568bcd49-db8st.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113412545Z namespaces/kserve/pods/seaweedfs-64568bcd49-db8st/seaweedfs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113420715Z namespaces/kserve/pods/seaweedfs-64568bcd49-db8st/seaweedfs/seaweedfs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113434856Z namespaces/kserve/pods/seaweedfs-64568bcd49-db8st/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113479249Z namespaces/kserve/pods/seaweedfs-64568bcd49-db8st/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113731366Z namespaces/kserve/pods/seaweedfs-64568bcd49-db8st/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113833763Z namespaces/kserve/pods/seaweedfs-64568bcd49-db8st/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113859325Z namespaces/kserve/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113914119Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.113985743Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.114044327Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.114139763Z namespaces/kuadrant-system/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.114198847Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.114282933Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.114301665Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.11437791Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.114421202Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.114503148Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.11498007Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115258098Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115312372Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115349365Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115418519Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115457302Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115538067Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115584691Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115613112Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.11572845Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115796705Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115834557Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.115895271Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.116016999Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.116133127Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.116250745Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.116315569Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.116372013Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.116559076Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.116734608Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.117185458Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.117262333Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.117653359Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.117864273Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118153003Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118283511Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118322674Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118430091Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118463263Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118520347Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118542858Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118631595Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118777224Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118843489Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118878111Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118924244Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.118987828Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.119072394Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.119098736Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.11916246Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.119536415Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.119846136Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.120012227Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.120238172Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.120439486Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.120477328Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-6qr6n.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.120822411Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.120869234Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.120961021Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121054797Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121159944Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121216628Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121284712Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121332185Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121370328Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121461944Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121620205Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121744473Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121830729Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.12184071Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.121887593Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino-7cf4765b85-5shw9.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.122001941Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.122010561Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.122020502Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.122101497Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.133308458Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.133411835Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-5shw9/authorino/authorino/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.133465919Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.133519352Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/authorino-operator-56fb6f6856-r8p5c.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.133671512Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.133685493Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.133693924Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.13377917Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134002114Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134102821Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-r8p5c/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134169586Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134246561Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/dns-operator-controller-manager-ccbdf686-nkbjm.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134357928Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134369359Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.13437863Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134548801Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134708982Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134812069Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-nkbjm/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134880644Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.134951218Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin-598f86468b-99wjj.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135056155Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135064966Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135080997Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135142971Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135243148Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135347095Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-99wjj/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135400308Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135457302Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/kuadrant-operator-controller-manager-66866f5fcc-4zxvt.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135615713Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135628154Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135634264Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.135712819Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146144968Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146217563Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-4zxvt/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146242125Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146297809Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador-limitador-85d9db4d5d-zllks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146386895Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146398055Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146406276Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146428187Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146514813Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146578988Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-zllks/limitador/limitador/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146623531Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146691515Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/limitador-operator-controller-manager-774d94476f-dzzbw.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146798262Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146809173Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146820124Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.146855496Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147004746Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.14706836Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-dzzbw/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147121734Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147160376Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.14721815Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147277954Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147353129Z namespaces/kube-node-lease/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.14736413Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.14736864Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147439475Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-29.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147567324Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-133-142.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147696922Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-135-177.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147767727Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147777638Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.147843062Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.148074218Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.148362237Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.148549489Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.148836799Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.149100406Z namespaces/kube-public/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.149108917Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.149117658Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.149171011Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.149329412Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.149513724Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.149724498Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.149968784Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.150259044Z namespaces/kube-system/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.150408814Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.150734906Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.150772958Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.150853024Z namespaces/kube-system/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.150894666Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151014585Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1510908Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151174255Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151226879Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151290703Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151338806Z namespaces/kube-system/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.15139524Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151475585Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151511798Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151553741Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151656178Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151704881Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151713872Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151743834Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-odkzeaftkrpzszjfowe5gtvoge.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.15184577Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.151929796Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.152029783Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.152233896Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.152324822Z namespaces/kube-system/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.152384957Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.152601901Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.152717789Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.153003118Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.153084533Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.153394724Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.153528973Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.153868106Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.15393835Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.153966563Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154046518Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154108162Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154200918Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154255302Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.15437103Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154460796Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154514889Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154564092Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154609086Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154648768Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154716533Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154725923Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154774566Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.154928157Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155103689Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155240458Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155439641Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155663586Z namespaces/kube-system/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155683807Z namespaces/kube-system/pods/global-pull-secret-syncer-jtb46/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155742971Z namespaces/kube-system/pods/global-pull-secret-syncer-jtb46/global-pull-secret-syncer-jtb46.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155827827Z namespaces/kube-system/pods/global-pull-secret-syncer-jtb46/global-pull-secret-syncer/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155838848Z namespaces/kube-system/pods/global-pull-secret-syncer-jtb46/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155844948Z namespaces/kube-system/pods/global-pull-secret-syncer-jtb46/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.155889531Z namespaces/kube-system/pods/global-pull-secret-syncer-jtb46/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156272797Z namespaces/kube-system/pods/global-pull-secret-syncer-jtb46/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156340842Z namespaces/kube-system/pods/global-pull-secret-syncer-jtb46/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156396285Z namespaces/kube-system/pods/global-pull-secret-syncer-rff9b/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156443449Z namespaces/kube-system/pods/global-pull-secret-syncer-rff9b/global-pull-secret-syncer-rff9b.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156514243Z namespaces/kube-system/pods/global-pull-secret-syncer-rff9b/global-pull-secret-syncer/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156522534Z namespaces/kube-system/pods/global-pull-secret-syncer-rff9b/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156527024Z namespaces/kube-system/pods/global-pull-secret-syncer-rff9b/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156559266Z namespaces/kube-system/pods/global-pull-secret-syncer-rff9b/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.156956393Z namespaces/kube-system/pods/global-pull-secret-syncer-rff9b/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157042119Z namespaces/kube-system/pods/global-pull-secret-syncer-rff9b/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157075711Z namespaces/kube-system/pods/global-pull-secret-syncer-vv2ch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157125684Z namespaces/kube-system/pods/global-pull-secret-syncer-vv2ch/global-pull-secret-syncer-vv2ch.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157192629Z namespaces/kube-system/pods/global-pull-secret-syncer-vv2ch/global-pull-secret-syncer/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157201199Z namespaces/kube-system/pods/global-pull-secret-syncer-vv2ch/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.15720524Z namespaces/kube-system/pods/global-pull-secret-syncer-vv2ch/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157236912Z namespaces/kube-system/pods/global-pull-secret-syncer-vv2ch/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157597096Z namespaces/kube-system/pods/global-pull-secret-syncer-vv2ch/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157677101Z namespaces/kube-system/pods/global-pull-secret-syncer-vv2ch/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157716974Z namespaces/kube-system/pods/konnectivity-agent-2ffrd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157749596Z namespaces/kube-system/pods/konnectivity-agent-2ffrd/konnectivity-agent-2ffrd.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157830571Z namespaces/kube-system/pods/konnectivity-agent-2ffrd/konnectivity-agent/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157837892Z namespaces/kube-system/pods/konnectivity-agent-2ffrd/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157850763Z namespaces/kube-system/pods/konnectivity-agent-2ffrd/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.157899256Z namespaces/kube-system/pods/konnectivity-agent-2ffrd/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1590021Z namespaces/kube-system/pods/konnectivity-agent-2ffrd/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.159070165Z namespaces/kube-system/pods/konnectivity-agent-2ffrd/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.159118798Z namespaces/kube-system/pods/konnectivity-agent-7qw2x/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.15915559Z namespaces/kube-system/pods/konnectivity-agent-7qw2x/konnectivity-agent-7qw2x.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.159242946Z namespaces/kube-system/pods/konnectivity-agent-7qw2x/konnectivity-agent/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.159252017Z namespaces/kube-system/pods/konnectivity-agent-7qw2x/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.159260837Z namespaces/kube-system/pods/konnectivity-agent-7qw2x/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.159310601Z namespaces/kube-system/pods/konnectivity-agent-7qw2x/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.160704004Z namespaces/kube-system/pods/konnectivity-agent-7qw2x/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.160771499Z namespaces/kube-system/pods/konnectivity-agent-7qw2x/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.160831512Z namespaces/kube-system/pods/konnectivity-agent-g5cl4/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.160872065Z namespaces/kube-system/pods/konnectivity-agent-g5cl4/konnectivity-agent-g5cl4.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.160958021Z namespaces/kube-system/pods/konnectivity-agent-g5cl4/konnectivity-agent/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.160971642Z namespaces/kube-system/pods/konnectivity-agent-g5cl4/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.160999024Z namespaces/kube-system/pods/konnectivity-agent-g5cl4/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.161009575Z namespaces/kube-system/pods/konnectivity-agent-g5cl4/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.162213765Z namespaces/kube-system/pods/konnectivity-agent-g5cl4/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.16228886Z namespaces/kube-system/pods/konnectivity-agent-g5cl4/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.162342824Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.162402868Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.162509395Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal/haproxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.162519316Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.162525556Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.16258209Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.162951294Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.16303761Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-29.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163110395Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163170319Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163260715Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal/haproxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163269626Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163277427Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1633334Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163586737Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163730767Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-142.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163789691Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163851825Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163954542Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal/haproxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163969383Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.163975573Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164021196Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164356299Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164446425Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164488477Z namespaces/kube-system/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164546981Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164606576Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164615616Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164708282Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164834721Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.164897315Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.16496856Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.165043785Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.165097218Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.165166373Z namespaces/observability/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.165178224Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.165188255Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.165244708Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.165437471Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.16571418Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.165878351Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.166121717Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.166375024Z namespaces/open-cluster-management-55c0f14d-689e-464a-b498-f413cbf2d/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.166382204Z namespaces/open-cluster-management-55c0f14d-689e-464a-b498-f413cbf2d/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.166392745Z namespaces/open-cluster-management-55c0f14d-689e-464a-b498-f413cbf2d/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.166457149Z namespaces/open-cluster-management-55c0f14d-689e-464a-b498-f413cbf2d/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1666218Z namespaces/open-cluster-management-55c0f14d-689e-464a-b498-f413cbf2d/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.166842915Z namespaces/open-cluster-management-55c0f14d-689e-464a-b498-f413cbf2d/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.166999736Z namespaces/open-cluster-management-55c0f14d-689e-464a-b498-f413cbf2d/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167229831Z namespaces/open-cluster-management-55c0f14d-689e-464a-b498-f413cbf2d/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167441975Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167450816Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167455696Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167497499Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167617067Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167686202Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167707993Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167780808Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.167946029Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.168145983Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.168300343Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.168527358Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.168784036Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.168925505Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169025192Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169081605Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169173522Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169226975Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169323032Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169437129Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169535956Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.16959313Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169648223Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169732589Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169778472Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169877559Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169936853Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.169999737Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.170094213Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.170164328Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.170225692Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.171212928Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.171301014Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.171440653Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.17154214Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.171653368Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.171817239Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172029743Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172118729Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172173592Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172250978Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172298831Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172381117Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1724324Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172541797Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172637014Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172814275Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172859198Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172934523Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.172992567Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173056472Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173100005Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173151468Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173158449Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173208002Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173363852Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173559476Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173758059Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.173981964Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.17437731Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174412593Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174465726Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174503539Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174610466Z namespaces/openshift-apiserver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174661349Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174763256Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174808859Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174856842Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174898625Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.174980391Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175061576Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175136361Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175178854Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175222987Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175286281Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175326574Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175408299Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175460763Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175496605Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175590752Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175693899Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.17571531Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175833308Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175920334Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.175994728Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176071194Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176151099Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176266887Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176432478Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176487161Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176523134Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176588048Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176625831Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176714697Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.17675848Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.176928741Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177046809Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177117414Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177175218Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177255233Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177296416Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.17736297Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177373421Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177437655Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177601186Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177818631Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.177970591Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178202457Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178534099Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178581462Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178685649Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178735392Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178803487Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178811808Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178822278Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178880002Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178959597Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.178970658Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.179042803Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.179203534Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.179390096Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.179552837Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.179808874Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.180016818Z namespaces/openshift-authentication/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.180032649Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18004632Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.180098734Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.180259664Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.180450727Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.180598647Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.180897487Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181115492Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181126072Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181130903Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181182376Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181264512Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181275413Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181335617Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181501197Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181713992Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.181865782Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182079846Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18228949Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182305511Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182310502Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182362205Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18243998Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182449651Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182495814Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18258616Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182602091Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182660895Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.182841237Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18302977Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1831815Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.183404785Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.183799932Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.183841065Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18392738Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.183970753Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184064009Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184106272Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184209669Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184305325Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184400192Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184457046Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184506189Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184577144Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184659549Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184756096Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184812859Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.184858753Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18496851Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185030404Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185051775Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18510977Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185216857Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185255399Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185373277Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185464013Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18556067Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185684668Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185778104Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.185948746Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186073704Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186132918Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186190912Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186260796Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18631618Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186378735Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186431898Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186518114Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186572848Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186695506Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186793242Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186854726Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18690735Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.186966964Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.187019667Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.187078921Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.187086162Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.187127145Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.187331098Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.187525161Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.187695233Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.187920518Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1881069Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188156603Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188223188Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188230459Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188264891Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188344046Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.18839586Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188463814Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188517528Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188615194Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188753834Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188870311Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.188924165Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189043643Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189078945Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189229335Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189325292Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189418308Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189477872Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189538966Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189620812Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189695047Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189799004Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189875169Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.189928232Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190026449Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190099474Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190106694Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190157888Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190264435Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190366412Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190509641Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190865365Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.190923669Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.191047377Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.191142594Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.191527329Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.191621716Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.191859992Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192037094Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192169972Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192228867Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19228507Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192356705Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192405918Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192475313Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192520976Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192589031Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192655005Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192757132Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192861169Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192918193Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.192992768Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.193052452Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.193121966Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.193207882Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.193217873Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.193287367Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.193494751Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.193702475Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.193860856Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194090161Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194295635Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194303006Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19436623Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/aws-ebs-csi-driver-node-6jmn4.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194483498Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-driver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194490838Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-driver/csi-driver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194500229Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-driver/csi-driver/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194561863Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194684631Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194776377Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19481367Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-liveness-probe/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19481976Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19482581Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.194896755Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195001012Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195086498Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195148452Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-node-driver-registrar/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195158463Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195162573Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195233828Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195348925Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195437862Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-6jmn4/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195497686Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1955636Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/aws-ebs-csi-driver-node-fwzwp.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195679638Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-driver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195689668Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-driver/csi-driver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19571589Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-driver/csi-driver/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195771464Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195887722Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.195980158Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19601684Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-liveness-probe/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196021921Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196027751Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196091275Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196210423Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196287759Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196335842Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-node-driver-registrar/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196343012Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196352003Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196421577Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196534675Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196618931Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fwzwp/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19675537Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196816194Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/aws-ebs-csi-driver-node-kr8kl.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19691212Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-driver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196932472Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-driver/csi-driver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196941942Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-driver/csi-driver/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.196992326Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197092472Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197174088Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197224751Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-liveness-probe/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197232832Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197238952Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197286385Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197366031Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197434785Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197481359Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-node-driver-registrar/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197493279Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.1975019Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197539652Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197632989Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197743836Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kr8kl/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19779502Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197840583Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197896816Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.197909607Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19795343Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198014804Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198064618Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198114891Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198166714Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198216478Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198223628Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198229749Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198277362Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198345657Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198355917Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198387039Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198446343Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198463994Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198506917Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.198889663Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.199075445Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.199216645Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19944736Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.19974896Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.199800714Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.199874849Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.199917402Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.199990607Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20003473Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200159928Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200235583Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200317489Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200412845Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200456168Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200541473Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200582036Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200628629Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200694324Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200793621Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200835853Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200853385Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200944171Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.200990374Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.201003584Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20123285Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.201311885Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.201357378Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.202306652Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.202407129Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.202568879Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.202656885Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203035741Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203188281Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203334301Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203405625Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203446888Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203502992Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203553055Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203608059Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203668633Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203730607Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203782671Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203864736Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.203945802Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204012246Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204044038Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204119293Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204157816Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204238581Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204279764Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204333708Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204382881Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204419353Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204443615Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204498389Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204739405Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.204953519Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.205205266Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.205437472Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.205661017Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.205679798Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-9chjn/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.205739482Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-9chjn/tuned-9chjn.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.205851219Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-9chjn/tuned/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20586105Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-9chjn/tuned/tuned/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.205869921Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-9chjn/tuned/tuned/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.205911824Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-9chjn/tuned/tuned/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206032332Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-9chjn/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206103556Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-9chjn/tuned/tuned/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206171031Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-shs4g/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206221364Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-shs4g/tuned-shs4g.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20630171Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-shs4g/tuned/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20631073Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-shs4g/tuned/tuned/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20631582Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-shs4g/tuned/tuned/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206349743Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-shs4g/tuned/tuned/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206470001Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-shs4g/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206544546Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-shs4g/tuned/tuned/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206578848Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zxmqg/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206621671Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zxmqg/tuned-zxmqg.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206747869Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zxmqg/tuned/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20675942Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zxmqg/tuned/tuned/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206765821Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zxmqg/tuned/tuned/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206814894Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zxmqg/tuned/tuned/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.206937152Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zxmqg/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207007687Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zxmqg/tuned/tuned/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207039629Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207085182Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207123005Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207166428Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20720827Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207218551Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207274745Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-29.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207369821Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-133-142.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207456107Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-135-177.ec2.internal.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20750249Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207544063Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207664771Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207723825Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.20780552Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207851203Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207912638Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.207996273Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208082029Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208218858Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208318605Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208372778Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208417111Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208477165Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208499597Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208586412Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208665168Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208718881Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208805137Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208881082Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.208929596Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209124459Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209206504Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209341883Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209417638Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209543067Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209703318Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209903331Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209957395Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.209990157Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.210440797Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21048674Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.210555295Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.210591587Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.210707965Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21078968Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.210845124Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.210880386Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211095511Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211124673Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211202028Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211312655Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211365539Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211409432Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211456275Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211463256Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211503308Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211654428Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.211890884Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212022123Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212226487Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212414849Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21242401Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212457132Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator-6748489656-p62z8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212556059Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator-watch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212563309Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212567479Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212605682Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212752782Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212834687Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212882571Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212892131Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212898101Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.212954675Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.213735218Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.213806913Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-p62z8/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.213842555Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.213897119Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.213961253Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.213992845Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214096932Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214140445Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214205039Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214239662Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214313767Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214351469Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214436445Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214541682Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21465857Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214730484Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214786958Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214844972Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.214881294Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21496594Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215009933Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215044116Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215128381Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215173034Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215190815Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215233628Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215318154Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215401639Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215449693Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21556323Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.215658097Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.216139679Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.216218304Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.216347093Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.216513094Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.216658213Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21675021Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.216794533Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.216859197Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.2169051Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.216982925Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217029429Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217087572Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217141306Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217220181Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217303187Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217362091Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217406224Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217460287Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217516861Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217604097Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217671401Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217717015Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217765928Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217850453Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217859704Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.217919178Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218074259Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218261511Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21839864Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218602504Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218812378Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218826639Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-klwbj/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218862471Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-klwbj/volume-data-source-validator-549c6d54cf-klwbj.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218942457Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-klwbj/volume-data-source-validator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218949887Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-klwbj/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.218954698Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-klwbj/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.2189892Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-klwbj/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219090587Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-klwbj/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219166962Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-klwbj/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219192903Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219235906Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21928555Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21929252Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219331753Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219388927Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.2194365Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219540057Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21957998Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219704648Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219837347Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.21989059Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219935663Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.219982406Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22002809Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220102904Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22018814Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220282387Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22034085Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220387834Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220444927Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22049221Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220571476Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22063125Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220693124Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.2207795Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220829563Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220836694Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220877076Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.220959692Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.221011435Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.221116162Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.221192977Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.221295654Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22137398Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.221450735Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.221585294Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.221895275Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22196922Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222013343Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222078917Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222119099Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222166943Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222211476Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222292101Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222373767Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222415259Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222452372Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222512976Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222555389Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222599472Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222612663Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222694908Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.222852229Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.223039981Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.223183571Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.223404556Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.223600529Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.223668863Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22376198Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.223799722Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.223910679Z namespaces/openshift-config-managed/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.223960563Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224036548Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224082121Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224168197Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224210119Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224319867Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224426284Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224549572Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224608526Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22466294Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224762937Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224798129Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224900336Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.224971411Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.225036385Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.225139732Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.225240589Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.225290812Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.228919255Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.229035533Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.22913982Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.229266328Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.229375156Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.229549777Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.229865549Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.229947374Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230021719Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230128106Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.23019242Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230251395Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230309158Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230357231Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230437527Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230506802Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230609719Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230745608Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230819422Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230858535Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230942741Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.230988264Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.231067379Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.23107888Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.231156345Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.231396221Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.231618326Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.231802488Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232024843Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232441351Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232495145Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232554349Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232564879Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232625014Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232732771Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232783474Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.23286689Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232897652Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232959166Z namespaces/openshift-config-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232970127Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.232992118Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.233042751Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.233126067Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.233137558Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.233211003Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.233369683Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.233563136Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.233733028Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.233980434Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234321717Z namespaces/openshift-config/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234384481Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234463857Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234521881Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234598316Z namespaces/openshift-config/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234680491Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234778758Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234878625Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.234981752Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235051076Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235118771Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235192786Z namespaces/openshift-config/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235226158Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235318424Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235360107Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235393899Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235477415Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235521158Z namespaces/openshift-config/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235568721Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235742412Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.235910284Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236031782Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236122418Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236223505Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236386036Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236522765Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236571958Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236616541Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236694916Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236737349Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236800023Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236838756Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.23689821Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.236938893Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.23704137Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237119075Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237164668Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237209141Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237264314Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237305477Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237357021Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237391183Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237436046Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237584176Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.237902227Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.238040646Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.2382451Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.238431433Z namespaces/openshift-config/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.238475435Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.238523419Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.238569532Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.23869534Z namespaces/openshift-console-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.238749354Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.238884713Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.238928766Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239026092Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239067565Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239150821Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239270639Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239373526Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239422069Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239462402Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239511155Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239550607Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239657665Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239714309Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239776263Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239861879Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239961615Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239969196Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.239995417Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.240068092Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.240114935Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.240740257Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.240823243Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241092221Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241244521Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241389151Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241514769Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241736424Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241797208Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241843291Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241891004Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241932057Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.241990251Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242038264Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24212072Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242196535Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24226851Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242307312Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242368827Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242406559Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242471473Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242504766Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24256284Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242591482Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242672527Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242704609Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242757302Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.242911563Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.243096815Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.243229614Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.243436898Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.243965034Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.243979184Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-rrhzz/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244005576Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-rrhzz/console-operator-584dcb6b54-rrhzz.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244114804Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-rrhzz/console-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244125144Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-rrhzz/console-operator/console-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244131054Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-rrhzz/console-operator/console-operator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244166117Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-rrhzz/console-operator/console-operator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244706403Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-rrhzz/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244756876Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244799479Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244855363Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244899716Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24495311Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24496291Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.244967081Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.245038915Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.245190576Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.245370048Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.245525698Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.245783235Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24600394Z namespaces/openshift-console/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.246050383Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.246248656Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.246277938Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.246359394Z namespaces/openshift-console/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.246387486Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.246482762Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.246657444Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.246991296Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24703964Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.247058181Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.247152467Z namespaces/openshift-console/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24719051Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.247279385Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.247330299Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.247369552Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.247470828Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.247529612Z namespaces/openshift-console/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.247571305Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.248216598Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.248308025Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.248580273Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24869094Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.248871552Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249020112Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249230606Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.2492855Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249319292Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249397147Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.24943099Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249505015Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249522086Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249615652Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249722669Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249780303Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249832407Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249868229Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249915272Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.249977556Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.250012129Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.250074403Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.250088084Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.250132167Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.250298258Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.250498171Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.250666212Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251062269Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251269093Z namespaces/openshift-console/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251288204Z namespaces/openshift-console/pods/console-76f56bf77c-cn6qr/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251329497Z namespaces/openshift-console/pods/console-76f56bf77c-cn6qr/console-76f56bf77c-cn6qr.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251431604Z namespaces/openshift-console/pods/console-76f56bf77c-cn6qr/console/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251443935Z namespaces/openshift-console/pods/console-76f56bf77c-cn6qr/console/console/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251449215Z namespaces/openshift-console/pods/console-76f56bf77c-cn6qr/console/console/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251457825Z namespaces/openshift-console/pods/console-76f56bf77c-cn6qr/console/console/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251573343Z namespaces/openshift-console/pods/console-76f56bf77c-cn6qr/console/console/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251659239Z namespaces/openshift-console/pods/console-76f56bf77c-cn6qr/console/console/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251709633Z namespaces/openshift-console/pods/downloads-7c97dc4555-m2kzp/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251743025Z namespaces/openshift-console/pods/downloads-7c97dc4555-m2kzp/downloads-7c97dc4555-m2kzp.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251849682Z namespaces/openshift-console/pods/downloads-7c97dc4555-m2kzp/download-server/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251860703Z namespaces/openshift-console/pods/downloads-7c97dc4555-m2kzp/download-server/download-server/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251865713Z namespaces/openshift-console/pods/downloads-7c97dc4555-m2kzp/download-server/download-server/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.251873073Z namespaces/openshift-console/pods/downloads-7c97dc4555-m2kzp/download-server/download-server/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252303682Z namespaces/openshift-console/pods/downloads-7c97dc4555-m2kzp/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252408849Z namespaces/openshift-console/pods/downloads-7c97dc4555-m2kzp/download-server/download-server/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252428991Z namespaces/openshift-console/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252482454Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252546349Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252598192Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252748052Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252796185Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.25286413Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252902082Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.252953786Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.25300748Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253100896Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253174351Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253259776Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253433068Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253501833Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253573777Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253603949Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253784461Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253848306Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253896959Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.253973614Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254045609Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254120294Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254237192Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254315687Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254394422Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254478768Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254559073Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254723054Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254923628Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.254975591Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255004773Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255071078Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255144372Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255216917Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255266351Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255348746Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255432172Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255486426Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255514767Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255571281Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255597413Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255652927Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255663307Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255720221Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.255958897Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.256140999Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.256284319Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.256495863Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.256725259Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.256762411Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.256823915Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.256871739Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.256967735Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257051151Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257109334Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257149777Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257380293Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257417815Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.25749818Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257574726Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257659771Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257780529Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257821072Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257888847Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.257930469Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258012335Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258057598Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.25809589Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258172315Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258226049Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258261552Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258296184Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.25838839Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258429113Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.25853579Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258611005Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258811708Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258891994Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.258968389Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.259108508Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.2592899Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.259527126Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.259568289Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.259620812Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.259688937Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.25973686Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.259778423Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.25988012Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.259983937Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26003808Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260096424Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260160199Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260206002Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260268006Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260275136Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260318969Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260500891Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260709876Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.260856905Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261083841Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261279034Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261306836Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261361809Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261405632Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261496768Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261558282Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261624837Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261774957Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261852112Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261884954Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.261979951Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262088298Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262172214Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262218277Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262261729Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262313563Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262350535Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26242132Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262483195Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262534578Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262605933Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26271233Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262740312Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26285708Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.262933505Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26301275Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263092135Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263176321Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263299519Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263518514Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263628941Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263709066Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263788672Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263844266Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26390985Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.263953433Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264041359Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264118664Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264165717Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264198099Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264251613Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264290245Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26435024Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264394812Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264447986Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264485189Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264531822Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264538862Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264564154Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264756377Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.264942109Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26509972Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.265571001Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.265814538Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.265857061Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.265947976Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26599775Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.266118538Z namespaces/openshift-dns/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.266165961Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.266232395Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.266258807Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267145157Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267204941Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267402444Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26748971Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267569105Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267615578Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267668522Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267749487Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26779195Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267895417Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.267969382Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.268022156Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26809396Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.268161485Z namespaces/openshift-dns/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.268209838Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.268402211Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.268512259Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.268788437Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.268864832Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.269162082Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.269301191Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.269513926Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26958602Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.269633754Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26972925Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.269773593Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.269839267Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.26988454Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.269995928Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270105005Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270194561Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270227483Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270284287Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27032363Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270373113Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270415196Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270461509Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270468759Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270495101Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270665973Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.270875037Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271016566Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271267303Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27151965Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271538301Z namespaces/openshift-dns/pods/dns-default-25cpf/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271604326Z namespaces/openshift-dns/pods/dns-default-25cpf/dns-default-25cpf.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271747425Z namespaces/openshift-dns/pods/dns-default-25cpf/dns/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271765586Z namespaces/openshift-dns/pods/dns-default-25cpf/dns/dns/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271777457Z namespaces/openshift-dns/pods/dns-default-25cpf/dns/dns/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271829241Z namespaces/openshift-dns/pods/dns-default-25cpf/dns/dns/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.271958389Z namespaces/openshift-dns/pods/dns-default-25cpf/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272055316Z namespaces/openshift-dns/pods/dns-default-25cpf/dns/dns/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272102919Z namespaces/openshift-dns/pods/dns-default-25cpf/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27211531Z namespaces/openshift-dns/pods/dns-default-25cpf/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272126021Z namespaces/openshift-dns/pods/dns-default-25cpf/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272197115Z namespaces/openshift-dns/pods/dns-default-25cpf/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27255613Z namespaces/openshift-dns/pods/dns-default-25cpf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272662397Z namespaces/openshift-dns/pods/dns-default-25cpf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272725861Z namespaces/openshift-dns/pods/dns-default-496j7/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272783185Z namespaces/openshift-dns/pods/dns-default-496j7/dns-default-496j7.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272869591Z namespaces/openshift-dns/pods/dns-default-496j7/dns/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272879711Z namespaces/openshift-dns/pods/dns-default-496j7/dns/dns/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272897242Z namespaces/openshift-dns/pods/dns-default-496j7/dns/dns/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.272933075Z namespaces/openshift-dns/pods/dns-default-496j7/dns/dns/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273041892Z namespaces/openshift-dns/pods/dns-default-496j7/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273129428Z namespaces/openshift-dns/pods/dns-default-496j7/dns/dns/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273189822Z namespaces/openshift-dns/pods/dns-default-496j7/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273198493Z namespaces/openshift-dns/pods/dns-default-496j7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273202673Z namespaces/openshift-dns/pods/dns-default-496j7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273247776Z namespaces/openshift-dns/pods/dns-default-496j7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273335452Z namespaces/openshift-dns/pods/dns-default-496j7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273405556Z namespaces/openshift-dns/pods/dns-default-496j7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27345579Z namespaces/openshift-dns/pods/dns-default-hg8ms/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273493582Z namespaces/openshift-dns/pods/dns-default-hg8ms/dns-default-hg8ms.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273585559Z namespaces/openshift-dns/pods/dns-default-hg8ms/dns/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273592929Z namespaces/openshift-dns/pods/dns-default-hg8ms/dns/dns/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273599049Z namespaces/openshift-dns/pods/dns-default-hg8ms/dns/dns/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273682525Z namespaces/openshift-dns/pods/dns-default-hg8ms/dns/dns/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273812154Z namespaces/openshift-dns/pods/dns-default-hg8ms/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273896949Z namespaces/openshift-dns/pods/dns-default-hg8ms/dns/dns/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273934692Z namespaces/openshift-dns/pods/dns-default-hg8ms/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273942642Z namespaces/openshift-dns/pods/dns-default-hg8ms/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.273966164Z namespaces/openshift-dns/pods/dns-default-hg8ms/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274017577Z namespaces/openshift-dns/pods/dns-default-hg8ms/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274135725Z namespaces/openshift-dns/pods/dns-default-hg8ms/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274229082Z namespaces/openshift-dns/pods/dns-default-hg8ms/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274271354Z namespaces/openshift-dns/pods/node-resolver-g9ssz/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274340699Z namespaces/openshift-dns/pods/node-resolver-g9ssz/node-resolver-g9ssz.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274458507Z namespaces/openshift-dns/pods/node-resolver-g9ssz/dns-node-resolver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274467678Z namespaces/openshift-dns/pods/node-resolver-g9ssz/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274476008Z namespaces/openshift-dns/pods/node-resolver-g9ssz/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274543752Z namespaces/openshift-dns/pods/node-resolver-g9ssz/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274772948Z namespaces/openshift-dns/pods/node-resolver-g9ssz/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274870165Z namespaces/openshift-dns/pods/node-resolver-g9ssz/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274932839Z namespaces/openshift-dns/pods/node-resolver-kvtkx/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.274976492Z namespaces/openshift-dns/pods/node-resolver-kvtkx/node-resolver-kvtkx.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275070968Z namespaces/openshift-dns/pods/node-resolver-kvtkx/dns-node-resolver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275078679Z namespaces/openshift-dns/pods/node-resolver-kvtkx/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275082689Z namespaces/openshift-dns/pods/node-resolver-kvtkx/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275134392Z namespaces/openshift-dns/pods/node-resolver-kvtkx/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275219228Z namespaces/openshift-dns/pods/node-resolver-kvtkx/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275278232Z namespaces/openshift-dns/pods/node-resolver-kvtkx/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275315654Z namespaces/openshift-dns/pods/node-resolver-mtzfw/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275350537Z namespaces/openshift-dns/pods/node-resolver-mtzfw/node-resolver-mtzfw.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275424962Z namespaces/openshift-dns/pods/node-resolver-mtzfw/dns-node-resolver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275432152Z namespaces/openshift-dns/pods/node-resolver-mtzfw/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275436162Z namespaces/openshift-dns/pods/node-resolver-mtzfw/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275478705Z namespaces/openshift-dns/pods/node-resolver-mtzfw/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275565691Z namespaces/openshift-dns/pods/node-resolver-mtzfw/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275624035Z namespaces/openshift-dns/pods/node-resolver-mtzfw/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275667698Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275726022Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275767305Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275838359Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.275944477Z namespaces/openshift-etcd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27600124Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276080686Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276112748Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276182292Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276214964Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276329342Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276431349Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276534486Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27659352Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276674245Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276733119Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276791913Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27689487Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.276965165Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.27703467Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277130436Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277207611Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277242953Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277352151Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277427336Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277510591Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277587147Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277699004Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.277839144Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278011485Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278073239Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278111812Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278162375Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278196777Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278242171Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278290764Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278401931Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278516459Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278590954Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278654088Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278735324Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278782127Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278842591Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278852851Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.278918276Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.279107569Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.279311552Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.279451632Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.279714729Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.279921223Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.279951205Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.28001989Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280058252Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280160329Z namespaces/openshift-host-network/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280208932Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280279877Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.28031986Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280384724Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280421546Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280516683Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280631471Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280764329Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280839005Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280901599Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.280981754Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281046638Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281160416Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281214159Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281261003Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281336618Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281425554Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281462996Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281606736Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281722304Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281800909Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281876324Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.281957739Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282127331Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282229607Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282275471Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282331704Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282388518Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282428251Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282491105Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282631445Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.282918544Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.28300517Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283117187Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283227065Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283288849Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283345333Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283399986Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283438219Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283491362Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283503373Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.283559907Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.28375874Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.284185809Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.284338299Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.284575245Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.284810211Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.284852434Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.284905787Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.2849441Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285057947Z namespaces/openshift-image-registry/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285115251Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285184186Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285234349Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285290793Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285336446Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285459554Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285618895Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.28585026Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285910184Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.285953157Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286006641Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286071955Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286216485Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286304351Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286386036Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286478282Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286546337Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286555868Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286617582Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286774072Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.286836916Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.287833363Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.288020476Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.288469706Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.288559932Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.28882887Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289011362Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289261449Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289337314Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289395548Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289452302Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289490604Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289552648Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289613482Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.28973039Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289836488Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289918613Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.289986378Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290072443Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290133367Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290260636Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290375713Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290460509Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290510153Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290598348Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290665393Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290728997Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.29077475Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290831214Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.290990075Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.291171007Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.291332938Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.291588865Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.291950379Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.29196406Z namespaces/openshift-image-registry/pods/image-registry-5b7ff5c8fb-j995z/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.292001742Z namespaces/openshift-image-registry/pods/image-registry-5b7ff5c8fb-j995z/image-registry-5b7ff5c8fb-j995z.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.29211654Z namespaces/openshift-image-registry/pods/image-registry-5b7ff5c8fb-j995z/registry/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.292123201Z namespaces/openshift-image-registry/pods/image-registry-5b7ff5c8fb-j995z/registry/registry/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.292129751Z namespaces/openshift-image-registry/pods/image-registry-5b7ff5c8fb-j995z/registry/registry/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.292183505Z namespaces/openshift-image-registry/pods/image-registry-5b7ff5c8fb-j995z/registry/registry/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.295810678Z namespaces/openshift-image-registry/pods/image-registry-5b7ff5c8fb-j995z/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.295881893Z namespaces/openshift-image-registry/pods/image-registry-5b7ff5c8fb-j995z/registry/registry/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.295923845Z namespaces/openshift-image-registry/pods/node-ca-4kw4j/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.295954987Z namespaces/openshift-image-registry/pods/node-ca-4kw4j/node-ca-4kw4j.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296119909Z namespaces/openshift-image-registry/pods/node-ca-4kw4j/node-ca/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296130469Z namespaces/openshift-image-registry/pods/node-ca-4kw4j/node-ca/node-ca/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296135399Z namespaces/openshift-image-registry/pods/node-ca-4kw4j/node-ca/node-ca/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296171252Z namespaces/openshift-image-registry/pods/node-ca-4kw4j/node-ca/node-ca/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296375255Z namespaces/openshift-image-registry/pods/node-ca-4kw4j/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.29644387Z namespaces/openshift-image-registry/pods/node-ca-4kw4j/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296475052Z namespaces/openshift-image-registry/pods/node-ca-986jm/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296515705Z namespaces/openshift-image-registry/pods/node-ca-986jm/node-ca-986jm.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.29673665Z namespaces/openshift-image-registry/pods/node-ca-986jm/node-ca/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296750321Z namespaces/openshift-image-registry/pods/node-ca-986jm/node-ca/node-ca/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296769002Z namespaces/openshift-image-registry/pods/node-ca-986jm/node-ca/node-ca/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.296818175Z namespaces/openshift-image-registry/pods/node-ca-986jm/node-ca/node-ca/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297192Z namespaces/openshift-image-registry/pods/node-ca-986jm/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297268135Z namespaces/openshift-image-registry/pods/node-ca-986jm/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297303728Z namespaces/openshift-image-registry/pods/node-ca-fl6jf/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.29734538Z namespaces/openshift-image-registry/pods/node-ca-fl6jf/node-ca-fl6jf.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297448207Z namespaces/openshift-image-registry/pods/node-ca-fl6jf/node-ca/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297459738Z namespaces/openshift-image-registry/pods/node-ca-fl6jf/node-ca/node-ca/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297512172Z namespaces/openshift-image-registry/pods/node-ca-fl6jf/node-ca/node-ca/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297558005Z namespaces/openshift-image-registry/pods/node-ca-fl6jf/node-ca/node-ca/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297802971Z namespaces/openshift-image-registry/pods/node-ca-fl6jf/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297877096Z namespaces/openshift-image-registry/pods/node-ca-fl6jf/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297911278Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.297946131Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298014365Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298049028Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298101401Z namespaces/openshift-infra/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298108462Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298112432Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298167106Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298322796Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.29852758Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298690781Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.298919336Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.299137051Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.299185304Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.299472553Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.299558569Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.299897062Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.299927534Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30017611Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300258346Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300340711Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300404516Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300444488Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300513113Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300552575Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300637011Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300717757Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30077006Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300848915Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300943182Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.300997195Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.301101712Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.301207159Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.301387742Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.301464276Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30180873Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30196833Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302185655Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302242599Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302282141Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302329825Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302366317Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302571921Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302617954Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302740652Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302916764Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.302965307Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30301352Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.303044273Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.303088925Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.303147539Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30315844Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.303219904Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.303371564Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.303562637Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.303856417Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.304240163Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.304896947Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.304916518Z namespaces/openshift-ingress-canary/pods/ingress-canary-6wjcw/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30494391Z namespaces/openshift-ingress-canary/pods/ingress-canary-6wjcw/ingress-canary-6wjcw.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305130982Z namespaces/openshift-ingress-canary/pods/ingress-canary-6wjcw/serve-healthcheck-canary/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305151714Z namespaces/openshift-ingress-canary/pods/ingress-canary-6wjcw/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305156724Z namespaces/openshift-ingress-canary/pods/ingress-canary-6wjcw/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30523666Z namespaces/openshift-ingress-canary/pods/ingress-canary-6wjcw/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305364028Z namespaces/openshift-ingress-canary/pods/ingress-canary-6wjcw/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305413171Z namespaces/openshift-ingress-canary/pods/ingress-canary-6wjcw/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305475725Z namespaces/openshift-ingress-canary/pods/ingress-canary-9r84r/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305503627Z namespaces/openshift-ingress-canary/pods/ingress-canary-9r84r/ingress-canary-9r84r.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305595704Z namespaces/openshift-ingress-canary/pods/ingress-canary-9r84r/serve-healthcheck-canary/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305605714Z namespaces/openshift-ingress-canary/pods/ingress-canary-9r84r/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305612565Z namespaces/openshift-ingress-canary/pods/ingress-canary-9r84r/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.30569675Z namespaces/openshift-ingress-canary/pods/ingress-canary-9r84r/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.305937496Z namespaces/openshift-ingress-canary/pods/ingress-canary-9r84r/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306024002Z namespaces/openshift-ingress-canary/pods/ingress-canary-9r84r/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306072596Z namespaces/openshift-ingress-canary/pods/ingress-canary-t22pn/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306109998Z namespaces/openshift-ingress-canary/pods/ingress-canary-t22pn/ingress-canary-t22pn.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306187483Z namespaces/openshift-ingress-canary/pods/ingress-canary-t22pn/serve-healthcheck-canary/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306201324Z namespaces/openshift-ingress-canary/pods/ingress-canary-t22pn/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306205494Z namespaces/openshift-ingress-canary/pods/ingress-canary-t22pn/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306256228Z namespaces/openshift-ingress-canary/pods/ingress-canary-t22pn/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306403618Z namespaces/openshift-ingress-canary/pods/ingress-canary-t22pn/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306476303Z namespaces/openshift-ingress-canary/pods/ingress-canary-t22pn/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306512275Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306561598Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306621722Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306700967Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306815995Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306862518Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306939083Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.306974186Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307045811Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307150898Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307230303Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307328689Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307548334Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307740727Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307769389Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307857235Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.307894438Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.308069899Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.308128343Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.308187977Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.308275703Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.308486247Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.308548601Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.309764593Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.309929574Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.310084534Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31017004Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.310341442Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.310477231Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.310752009Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.310829404Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.310876177Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.310960803Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311004716Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31106807Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311078301Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311166087Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311239802Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311280654Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311402903Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311496979Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311567654Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311616227Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311704843Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311741155Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311829721Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.311993682Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312139602Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312217867Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312279321Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312287062Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312415581Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312496446Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312506096Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31256486Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312755453Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.312967827Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.313165591Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.313424298Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.313659864Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.313716208Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.313782212Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.313817494Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.313926322Z namespaces/openshift-ingress/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.313963694Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31404549Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.314080602Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.314149067Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.314778659Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.315019405Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.315280202Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.315518898Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.315579182Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.315624165Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.315713642Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31584295Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.316020732Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.316080196Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31613934Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.316363745Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.316943904Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.316966546Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.317064262Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31718102Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.317265635Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.317446218Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.317582497Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.318024966Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.318126983Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.318341298Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.318416503Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.318733364Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.318870073Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319113249Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319241048Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319293641Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31941665Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319448022Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319456092Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.31957044Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319619013Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319733751Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319789595Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.319886931Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320001369Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320075564Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320127227Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320188011Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320250975Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320342372Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320392375Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320453399Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320499032Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320566357Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320593649Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320651822Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.320834675Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321030448Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321169287Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321371441Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321582145Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321590395Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321624427Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/istiod-openshift-gateway-5677ccb49b-82lr6.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321798739Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321818591Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321823951Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.321849893Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.331765657Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.331843783Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-82lr6/discovery/discovery/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.331919108Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.331994062Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.332128342Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.332137542Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.332141832Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.332204616Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.333560117Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-pfnkg/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.333658424Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.333790543Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router-default-5657cc6765-srj79.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.333920122Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.333928732Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.333941143Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.333987906Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334084642Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334163678Z namespaces/openshift-ingress/pods/router-default-5657cc6765-srj79/router/router/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334213331Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334257344Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334302437Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334364161Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334478229Z namespaces/openshift-insights/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334512821Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334604017Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334622998Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334733306Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334755078Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.334904857Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335027716Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335128872Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335159955Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335210038Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335286973Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335322136Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335401651Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335436183Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335519389Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335575933Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335624446Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335727133Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335811738Z namespaces/openshift-insights/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.335845761Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.336995828Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.337108835Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.337400885Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.337537094Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.337864246Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.337954782Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338111772Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338179457Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.33822891Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338292355Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338340408Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338434154Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338503839Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338542871Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338618946Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338738544Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338797178Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338844761Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.338910156Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3389639Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339038225Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339112439Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339174794Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339213776Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339280481Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339326434Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339405939Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339440492Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339511786Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339540278Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339606272Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339616413Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339684288Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.339898682Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.340108246Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.340261246Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.340487102Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34076323Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.340776341Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-msvxc/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.340804163Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-msvxc/insights-operator-5885fdb759-msvxc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34091984Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-msvxc/insights-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.340927611Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-msvxc/insights-operator/insights-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.340934221Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-msvxc/insights-operator/insights-operator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.341002876Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-msvxc/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.341426194Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-msvxc/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.341778808Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.341812391Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/insights-runtime-extractor-czx2d.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.341948089Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.341961621Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/exporter/exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.341966661Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/exporter/exporter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.341975081Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/exporter/exporter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342081929Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342151053Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/exporter/exporter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342178995Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/extractor/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342185496Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/extractor/extractor/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342190096Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/extractor/extractor/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342228018Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/extractor/extractor/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342333405Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34240624Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/extractor/extractor/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342443883Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342453854Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342462104Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342514867Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342611074Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3427089Z namespaces/openshift-insights/pods/insights-runtime-extractor-czx2d/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342770004Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342829348Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/insights-runtime-extractor-nvtkq.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342919485Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342929125Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/exporter/exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.342934916Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/exporter/exporter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34299694Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/exporter/exporter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343095906Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34315771Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/exporter/exporter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343185543Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/extractor/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343194823Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/extractor/extractor/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343209124Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/extractor/extractor/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343254307Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/extractor/extractor/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343349583Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343428739Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/extractor/extractor/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34344058Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343506834Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343514845Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343560137Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343668695Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343784432Z namespaces/openshift-insights/pods/insights-runtime-extractor-nvtkq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343826175Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343863428Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/insights-runtime-extractor-shhb8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343953434Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343961664Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/exporter/exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.343974965Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/exporter/exporter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344010968Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/exporter/exporter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344100984Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344167408Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/exporter/exporter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34420178Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/extractor/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344212631Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/extractor/extractor/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344222182Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/extractor/extractor/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344276925Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/extractor/extractor/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344368632Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344431256Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/extractor/extractor/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344447087Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344453417Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344460778Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344565245Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344682413Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344765118Z namespaces/openshift-insights/pods/insights-runtime-extractor-shhb8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34478585Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344841873Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344883616Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344924919Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344985683Z namespaces/openshift-keda/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.344994834Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345006874Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345043437Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345137403Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345195597Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345204618Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34523666Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345293804Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345335876Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345415962Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345500807Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345561002Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345570102Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345627486Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.345798407Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34598294Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.346202945Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.346411529Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.346568209Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.346763862Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.346796574Z namespaces/openshift-keda/operators.coreos.com/installplans/install-7vjt5.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.346919043Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.346970626Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34702254Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347064322Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347123426Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347166779Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347249685Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347292338Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347359042Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347381174Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347444448Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347485791Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347567376Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347654952Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347750868Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347806042Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347838554Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347892128Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.347936861Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348015936Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34806753Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348099562Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348188078Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348231721Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348278174Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348380971Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348478167Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348553732Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348631867Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348743425Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.348864603Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349047365Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349112109Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349144212Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349196925Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349233458Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349288861Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349320083Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349403149Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349490465Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349538548Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349583861Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349623924Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349681338Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.34972283Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349731251Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349775234Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.349934905Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.350113317Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.350254086Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35045507Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.350663264Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.350731678Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.350787242Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.350823354Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.350896229Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.350937032Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351002216Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351044249Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351113944Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351129575Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351224151Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351309307Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351386042Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351420364Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351726515Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351738856Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351743136Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351746976Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351750906Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351760977Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351842212Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.351907207Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3519515Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352059537Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352142773Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352217458Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352290563Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352364338Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352498636Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352685599Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352735823Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352771605Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352825058Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352879292Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352913754Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.352948287Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353048713Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353122288Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353172642Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353204524Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35330081Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353322202Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353455501Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353535866Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353571788Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353627522Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35374536Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353772302Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353779202Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.353882189Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35403785Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.354225922Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.354369022Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.354564835Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.354775619Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.354813002Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.354869625Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.354919059Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.354987703Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355035437Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355097191Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355135373Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355195447Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35523711Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355326356Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355403241Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355481486Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355518749Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355562732Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355619946Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3556901Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355793807Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3558378Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355882643Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.355975629Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.356051574Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.356105868Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.356256858Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.356454052Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.356544587Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.356614162Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35673029Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.356855098Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357060002Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357132717Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35717785Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357229893Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357279947Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35732654Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357373693Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357450828Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357528313Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357583007Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357631311Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357730927Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35776776Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357836484Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357871107Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35792694Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357941131Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.357985904Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.358138794Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.358326867Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.358465316Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.358794349Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359000762Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359029634Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359099729Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359136781Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359239768Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359285051Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359376728Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.35941777Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359476074Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359523317Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359610693Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359726291Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359816827Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3598684Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359914473Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.359961847Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360003469Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360086045Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360135908Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360183891Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360267837Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360319631Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360327131Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360364924Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360426598Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360471221Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360566067Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.360663894Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.361416164Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.36151093Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.361622288Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.361794039Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362008464Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362079478Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362136442Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362211967Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362261371Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362336616Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362381389Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362473255Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362560441Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362609764Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362655967Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362732372Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362765615Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362825789Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362835059Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.362862421Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.363039723Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.363247417Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.363409378Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.363670585Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.363875299Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.363905281Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.363974436Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364011198Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364076752Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364123755Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3641868Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364218652Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364275886Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364307238Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364394754Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364472759Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364549684Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364598927Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364663322Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364798291Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364840903Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364916999Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.364964192Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365024236Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365104901Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365161115Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365202208Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365310515Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.36538272Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365457485Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.36553174Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365615386Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365771226Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.365943617Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3659897Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366037724Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366085987Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366118239Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366183153Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366222116Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366308232Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366390187Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366442291Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366474423Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366527536Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.36657069Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366607152Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366629443Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366696228Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.366862919Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.367053522Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.367413516Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.367654042Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.367873857Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.36791148Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.367986134Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368046109Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368123714Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368187248Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368244692Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368296955Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368360739Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368509Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368621117Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368744795Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368828971Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368889765Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.368941898Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369020784Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369059336Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369162393Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369214467Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.36926126Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369332075Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369398629Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369455023Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.36955892Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369633995Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369751383Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369896602Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.369973558Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370100366Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370298429Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370347492Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370402286Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370451719Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370510423Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370570918Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370626561Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370738069Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370826915Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370886519Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370927571Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.370986445Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.371038399Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.371101333Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.371119954Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.371161367Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.371341419Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.371545323Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.371747266Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.371961701Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372176405Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372224468Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372292763Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372343306Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372428892Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372494667Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372569701Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372601034Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372773025Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372825209Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.372921745Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373095637Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373210404Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373266338Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373312191Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373379136Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373428769Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373538946Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373578149Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373628223Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373753341Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373792594Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373806335Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373852558Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373918192Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.373961455Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.374071762Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.374160918Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.37433788Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.374412365Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.374534153Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.374679043Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375097601Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375162035Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375208288Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375272853Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375327666Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.37538386Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375438124Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3755382Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375621336Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375704591Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375752095Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375801498Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375896094Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375951288Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375958579Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.375993781Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.376150501Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.376366236Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.376529747Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.376785694Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.376994818Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377002358Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377054172Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.37717838Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s/kube-storage-version-migrator-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377189701Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377199062Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377242125Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377388054Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377534774Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377580137Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377636491Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377698705Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377746478Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377810513Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377857986Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377905079Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.377961923Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378004746Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378091582Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378210559Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378346579Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378409083Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378462086Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.37851673Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378568243Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378679911Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378754996Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378797969Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.378899766Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.37895644Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.379002803Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.379125771Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.379219587Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.379357206Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.379449943Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.379580731Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.379818757Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3800106Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380073234Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380121888Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380186422Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380234355Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380296429Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380353563Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38044896Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380538955Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38059691Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380651263Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380729058Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380788712Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380856807Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380864817Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.380915961Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.381106264Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.381317728Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.381479718Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.381744726Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.381961921Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.381973982Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382018905Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/migrator-6865c9967-zqrks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382156174Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/graceful-termination/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382163574Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/graceful-termination/graceful-termination/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382173325Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382292663Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382406171Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382490776Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382536459Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/migrator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38254491Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/migrator/migrator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38254881Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/migrator/migrator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382594153Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/migrator/migrator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382738803Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382825919Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-zqrks/migrator/migrator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382865081Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382925806Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.382962638Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383020802Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383092517Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38314763Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383228386Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383278589Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383338683Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383386677Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383490413Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383703818Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383839247Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383902041Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.383954815Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384013239Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384096434Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384196711Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384245604Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384408115Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384514942Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384568876Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384576426Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38462548Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384757828Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384833763Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.384882527Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.385035627Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.385158875Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38552606Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.385626607Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.385863022Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386010752Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386237137Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386311882Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386348275Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386425Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386461402Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386525377Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386562029Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386665886Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386750942Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386804055Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386844078Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386902822Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.386950355Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.387060923Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.387118847Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38716987Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.387181811Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.387223243Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.387376444Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.387572667Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.387820394Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.388095732Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.388293815Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.388734845Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.388757296Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-9vllq.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.388857073Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.388911127Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.388988012Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389047846Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38911031Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389156623Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389233298Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389247889Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-n59f7/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389293462Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-n59f7/lws-controller-manager-fd6565d47-n59f7.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389392519Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-n59f7/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38940146Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-n59f7/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.38940894Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-n59f7/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389492646Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-n59f7/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389657667Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-n59f7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389756753Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-n59f7/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389828428Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-w5jsc/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.389899863Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-w5jsc/lws-controller-manager-fd6565d47-w5jsc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.3900058Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-w5jsc/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390017811Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-w5jsc/manager/manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390088455Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-w5jsc/manager/manager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390102347Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-w5jsc/manager/manager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390473291Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-w5jsc/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390547766Z namespaces/openshift-lws-operator/pods/lws-controller-manager-fd6565d47-w5jsc/manager/manager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.39059862Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-z4vbw/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390678905Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-z4vbw/openshift-lws-operator-fd8ccff4c-z4vbw.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390774252Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-z4vbw/openshift-lws-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390781932Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-z4vbw/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390788612Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-z4vbw/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.390839736Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-z4vbw/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391017378Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-z4vbw/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391085772Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-z4vbw/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391114364Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391157607Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.39120627Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391253184Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391302987Z namespaces/openshift-machine-api/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391311308Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391315258Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.39135686Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391528022Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391735166Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.391960641Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392205147Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392413201Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392421752Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392425772Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392483526Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392654287Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392763925Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392772355Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.392827429Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.393022332Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.393231686Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.393396347Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.393672446Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.39388747Z namespaces/openshift-marketplace/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.393897711Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.393902951Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.393944194Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.39403665Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.394050451Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.394081033Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.394190741Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.394292737Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.394395524Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.394460239Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.394510542Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.394707565Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.39493798Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.395100291Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.395337587Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.395573313Z namespaces/openshift-monitoring/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.395657199Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.395835601Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.395875173Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.395966659Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.395999182Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.396177694Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.396692458Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397126497Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397400936Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397440818Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397510583Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397567577Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397691085Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397821604Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397868437Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.397974114Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.398026497Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.398077341Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.401880606Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.402055087Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.403272979Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.403366265Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.404544984Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.404717056Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.405286934Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.405698561Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.405749635Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.405929907Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.40596848Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406038264Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406083357Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406218807Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406280411Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406360136Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406410589Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406544428Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406577591Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406710439Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406757443Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406896382Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.406947245Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.407077354Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.407279098Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.407392915Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.407732958Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.407899169Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.40805515Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.408165057Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.408289465Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.408398353Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.408503549Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.408555793Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.408803549Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.408913877Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.409029295Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.409150483Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.40925722Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.409434942Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.409547639Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.409663627Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.409799106Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.409901233Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41000508Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.410092016Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41014576Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.410210264Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.410217484Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.410269048Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.410527975Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.410819095Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.410998127Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.411242703Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.411510441Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.411555924Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.411761818Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.411779619Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.411861285Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412052347Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412061958Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412069848Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412121852Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41223689Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412433303Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412489487Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412499987Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412506928Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412561481Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412700881Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412790987Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41284305Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412850371Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.412856401Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413026993Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41313576Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413219705Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413269889Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413278189Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41328424Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413333563Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413455141Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413539357Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41358641Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413594801Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413610802Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413648654Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413784273Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413870639Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413925183Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413933383Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.413947954Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414000468Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414125306Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414213432Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414270136Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414279356Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414285507Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414357042Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41447252Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414553705Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414605668Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9mw5h/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414666202Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9mw5h/cluster-monitoring-operator-548b489cb8-9mw5h.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414803512Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9mw5h/cluster-monitoring-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414817552Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9mw5h/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414828373Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9mw5h/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.414853275Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9mw5h/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.415790578Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9mw5h/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.415877574Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9mw5h/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.415956479Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416009463Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-state-metrics-56d9f944b4-kxrdb.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416160933Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-main/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416174564Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416183764Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416242608Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416359976Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416442112Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416495395Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-self/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416503566Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416511746Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416584951Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41671968Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416805466Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416852699Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-state-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41686313Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416873301Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.416917813Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417209713Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4173167Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-kxrdb/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417361653Z namespaces/openshift-monitoring/pods/metrics-server-789948b4bd-tbvt4/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417419417Z namespaces/openshift-monitoring/pods/metrics-server-789948b4bd-tbvt4/metrics-server-789948b4bd-tbvt4.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417521574Z namespaces/openshift-monitoring/pods/metrics-server-789948b4bd-tbvt4/metrics-server/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417532465Z namespaces/openshift-monitoring/pods/metrics-server-789948b4bd-tbvt4/metrics-server/metrics-server/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417542585Z namespaces/openshift-monitoring/pods/metrics-server-789948b4bd-tbvt4/metrics-server/metrics-server/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417595059Z namespaces/openshift-monitoring/pods/metrics-server-789948b4bd-tbvt4/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41776531Z namespaces/openshift-monitoring/pods/metrics-server-789948b4bd-tbvt4/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417859587Z namespaces/openshift-monitoring/pods/metrics-server-789948b4bd-tbvt4/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417902319Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-m9dql/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.417961433Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-m9dql/monitoring-plugin-68b7fc9574-m9dql.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41806573Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-m9dql/monitoring-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418074071Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-m9dql/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418078071Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-m9dql/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418140795Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-m9dql/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418266094Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-m9dql/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418346939Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-m9dql/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418407123Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418467477Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/node-exporter-4nmcq.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418603076Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/init-textfile/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418612957Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/init-textfile/init-textfile/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418623808Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/init-textfile/init-textfile/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418683252Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418785669Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418874024Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418932568Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418942439Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41895246Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.418999713Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4191111Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419212597Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419276861Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/node-exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419289712Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/node-exporter/node-exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419295943Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/node-exporter/node-exporter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419364837Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419504057Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419583392Z namespaces/openshift-monitoring/pods/node-exporter-4nmcq/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.41970789Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419758074Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/node-exporter-cbmtz.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419888572Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/init-textfile/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419898743Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/init-textfile/init-textfile/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419903293Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/init-textfile/init-textfile/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.419961377Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420072745Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42015461Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420204484Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420211634Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420215604Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420266118Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420376105Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420458501Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420513484Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/node-exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420520685Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/node-exporter/node-exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420528105Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/node-exporter/node-exporter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420582099Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42074786Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420832156Z namespaces/openshift-monitoring/pods/node-exporter-cbmtz/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42088715Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.420949114Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/node-exporter-g2cmr.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421069032Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/init-textfile/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421078482Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/init-textfile/init-textfile/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421084793Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/init-textfile/init-textfile/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421128195Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421217061Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421300477Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42135075Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421361381Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421371412Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421415595Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421529842Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421609358Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421655261Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/node-exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421670362Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/node-exporter/node-exporter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421679972Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/node-exporter/node-exporter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421745867Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.421868775Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42194634Z namespaces/openshift-monitoring/pods/node-exporter-g2cmr/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422007494Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422063868Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/openshift-state-metrics-b8f49f5c6-nwgmw.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42223578Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-main/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42224273Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422266242Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422329576Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422454964Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422553901Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422596624Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-self/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422605174Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422614755Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42269211Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422820519Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.422957628Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423002481Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/openshift-state-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423009441Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423021572Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423077676Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423206825Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423299611Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-nwgmw/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423372586Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423464422Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.423981467Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424014869Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424019969Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42418756Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424297848Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424378853Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424424106Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424429347Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424452468Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424510062Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424674233Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424792961Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424840664Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424850105Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424864386Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.424908399Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425039588Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425132864Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425173077Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425179597Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425188738Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425261903Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42537308Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425469526Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425512189Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42552317Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425533321Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425582334Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425715613Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425802379Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425853132Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425863303Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425874794Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.425918327Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.426788625Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.426882661Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.426917803Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.426930975Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.426936485Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427023981Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427151599Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427256326Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427301779Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427372384Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/prometheus-operator-84d69f794d-f2z88.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427514494Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427524104Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427529004Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427573098Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427776091Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427884428Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42790545Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/prometheus-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427916761Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427932802Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.427991435Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42835814Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.428445296Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-f2z88/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.428490959Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-zkptp/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.428562634Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-zkptp/prometheus-operator-admission-webhook-7f8d78f54b-zkptp.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.428745536Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-zkptp/prometheus-operator-admission-webhook/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.428757677Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-zkptp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.428769427Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-zkptp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.428825271Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-zkptp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.428940369Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-zkptp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429061667Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-zkptp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429132872Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429197036Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/telemeter-client-5478597cd8-pw7rm.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429384509Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429396809Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4294019Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429440343Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.42954923Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429635225Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429711361Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/reload/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429721301Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/reload/reload/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429729392Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/reload/reload/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429779455Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/reload/reload/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429889133Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.429971418Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/reload/reload/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430023602Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/telemeter-client/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430039543Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/telemeter-client/telemeter-client/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430045473Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430113078Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430384526Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430472112Z namespaces/openshift-monitoring/pods/telemeter-client-5478597cd8-pw7rm/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430505004Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43059105Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/thanos-querier-cdb8d69-rlllz.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430833466Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430854047Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430861508Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.430877859Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431000267Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431076332Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43119708Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-rules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431204501Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431211261Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431268225Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431396954Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43149546Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431537193Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-web/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431546724Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431551514Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431615058Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431766179Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431834633Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431882906Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431890897Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431898197Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.431957191Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432074089Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432163425Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432216989Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/prom-label-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432225779Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43223178Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432282803Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432400181Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432490287Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43254086Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/thanos-query/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432549281Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/thanos-query/thanos-query/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432555791Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/thanos-query/thanos-query/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432620056Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.432906945Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433024153Z namespaces/openshift-monitoring/pods/thanos-querier-cdb8d69-rlllz/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433062355Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43312449Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433187944Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433245818Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433628793Z namespaces/openshift-multus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433709669Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433815856Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433867719Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433934424Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.433984847Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.434258585Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.434364822Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43447038Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.434525273Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.434580597Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.434812703Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.434954262Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.435035037Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.435083371Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.435152535Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.435205889Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.435307886Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43536606Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.435418223Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43552546Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.435596485Z namespaces/openshift-multus/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.435656369Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4359635Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.436067987Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.436688318Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.436850729Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43760595Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.437736659Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.437901599Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.437980835Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.438040619Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.438385632Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43850172Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.438618538Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.438769108Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43880945Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.438971501Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439074958Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439185185Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.43925881Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439319705Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439390229Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439440783Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439536849Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439585182Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439658687Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439724222Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439817038Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439906714Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.439972588Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440018131Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440101047Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440165221Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440236096Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440280769Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440341503Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440350644Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440414298Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440601771Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.440871518Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44104239Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.441284236Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.441501021Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.441509681Z namespaces/openshift-multus/pods/multus-72bsm/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.441561495Z namespaces/openshift-multus/pods/multus-72bsm/multus-72bsm.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.441701424Z namespaces/openshift-multus/pods/multus-72bsm/kube-multus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.441715385Z namespaces/openshift-multus/pods/multus-72bsm/kube-multus/kube-multus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.441720026Z namespaces/openshift-multus/pods/multus-72bsm/kube-multus/kube-multus/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.441799661Z namespaces/openshift-multus/pods/multus-72bsm/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.442541721Z namespaces/openshift-multus/pods/multus-72bsm/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.442623316Z namespaces/openshift-multus/pods/multus-72bsm/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.442715642Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.442786477Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/multus-additional-cni-plugins-dxvgt.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.442964459Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/bond-cni-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44297602Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44298218Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443027303Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4431284Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443215276Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443253318Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443261279Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/cni-plugins/cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443267759Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443321703Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443437401Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443543358Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44358358Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/egress-router-binary-copy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443589431Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443669426Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443819516Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.443939614Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444039211Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444077913Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444083694Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444089974Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44418093Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444287818Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444376384Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444421556Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/routeoverride-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444435017Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444441388Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444567886Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444715996Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444827494Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444872117Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni-bincopy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444883847Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444890118Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.444938651Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44506796Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445165136Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44522383Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445231761Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445235671Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445280104Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445393881Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445496879Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-dxvgt/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445542131Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445599895Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/multus-additional-cni-plugins-ljfhd.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.445993662Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/bond-cni-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446004762Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446008793Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446050116Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446164483Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446252809Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446299862Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446306973Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/cni-plugins/cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446310883Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446355446Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446468724Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446539738Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446578311Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/egress-router-binary-copy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446583211Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446587342Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446671017Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446781295Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446869211Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446910373Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446919544Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446923564Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.446984908Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447105276Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447140859Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447197062Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/routeoverride-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447205523Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447209623Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447246056Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447361044Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44745892Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447487452Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni-bincopy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447494953Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447501973Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447585479Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447690586Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447779452Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447805983Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447812384Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447818174Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447866137Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.447975885Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448074911Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-ljfhd/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448219861Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448232762Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/multus-additional-cni-plugins-lr2pj.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44835582Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/bond-cni-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44836056Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448371721Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448423485Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448536432Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448632879Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448699293Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448722125Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/cni-plugins/cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448729095Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448769048Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448871945Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448934109Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448966881Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/egress-router-binary-copy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448988742Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.448993093Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449040526Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449132302Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449200517Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449236369Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449243419Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44924737Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449303624Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44940242Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449495196Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44954831Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/routeoverride-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449555331Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449561341Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449607914Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449740123Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449809088Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449836919Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni-bincopy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44984165Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.44984719Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449896783Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.449980589Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450101367Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450110638Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450114558Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450121228Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.45014578Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450248427Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450332332Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-lr2pj/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450371035Z namespaces/openshift-multus/pods/multus-st8s2/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450427629Z namespaces/openshift-multus/pods/multus-st8s2/multus-st8s2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450562988Z namespaces/openshift-multus/pods/multus-st8s2/kube-multus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450573569Z namespaces/openshift-multus/pods/multus-st8s2/kube-multus/kube-multus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450578889Z namespaces/openshift-multus/pods/multus-st8s2/kube-multus/kube-multus/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.450587189Z namespaces/openshift-multus/pods/multus-st8s2/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.451604118Z namespaces/openshift-multus/pods/multus-st8s2/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.451814662Z namespaces/openshift-multus/pods/multus-st8s2/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.451865085Z namespaces/openshift-multus/pods/multus-zcfck/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.451916649Z namespaces/openshift-multus/pods/multus-zcfck/multus-zcfck.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.452006025Z namespaces/openshift-multus/pods/multus-zcfck/kube-multus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.452015215Z namespaces/openshift-multus/pods/multus-zcfck/kube-multus/kube-multus/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.452019166Z namespaces/openshift-multus/pods/multus-zcfck/kube-multus/kube-multus/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.452070599Z namespaces/openshift-multus/pods/multus-zcfck/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.452818689Z namespaces/openshift-multus/pods/multus-zcfck/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453016063Z namespaces/openshift-multus/pods/multus-zcfck/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453038394Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453048214Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/network-metrics-daemon-89z9z.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453169303Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453180354Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453187834Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453237187Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453374916Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453462492Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453511766Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/network-metrics-daemon/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453519006Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453525486Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.45358551Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453826417Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453918273Z namespaces/openshift-multus/pods/network-metrics-daemon-89z9z/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.453965066Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454037271Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/network-metrics-daemon-n8rn6.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454150248Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454157199Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454161189Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454201212Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.45432488Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454425447Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454457079Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/network-metrics-daemon/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454462269Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.45447226Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454535394Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454741398Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454809903Z namespaces/openshift-multus/pods/network-metrics-daemon-n8rn6/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454851445Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454877837Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/network-metrics-daemon-sjtrc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454962123Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454969173Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454973354Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.454996905Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455090601Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455168236Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455204619Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/network-metrics-daemon/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455209389Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.45521519Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455259393Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455461946Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455620817Z namespaces/openshift-multus/pods/network-metrics-daemon-sjtrc/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455675941Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455709633Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455763596Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455770637Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455804829Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.455898205Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.45596151Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456002823Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456090208Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456137492Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456176274Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456222317Z namespaces/openshift-must-gather-rd9v9/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456236648Z namespaces/openshift-must-gather-rd9v9/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456243569Z namespaces/openshift-must-gather-rd9v9/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456274071Z namespaces/openshift-must-gather-rd9v9/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.456485505Z namespaces/openshift-must-gather-rd9v9/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4567169Z namespaces/openshift-must-gather-rd9v9/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.45685843Z namespaces/openshift-must-gather-rd9v9/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457078945Z namespaces/openshift-must-gather-rd9v9/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457275608Z namespaces/openshift-network-console/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457321331Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457386705Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457424238Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.45745472Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457507143Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457582058Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457724018Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457920391Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.457967584Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458008937Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458081142Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458122285Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458167058Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4582078Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458288776Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458333969Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458382562Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458465217Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458511771Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458554974Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458685712Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458788499Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458907147Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.458982962Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459132292Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459260951Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459384999Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459442743Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459479465Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459535799Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459573952Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459627495Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459751194Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459806878Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4598431Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459886663Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.459919875Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460003061Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460094087Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46013774Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460183493Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460229326Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460262788Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460314052Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460321212Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460350004Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460504214Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460705008Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.460844967Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461083023Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461277336Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461284646Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-zb6h5/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461311618Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-zb6h5/networking-console-plugin-74c9f884b7-zb6h5.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461403874Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-zb6h5/networking-console-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461412885Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-zb6h5/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461417905Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-zb6h5/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461448408Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-zb6h5/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461553635Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-zb6h5/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461602848Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-zb6h5/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4616372Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461700604Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461754938Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461792251Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461851535Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461894297Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461955702Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.461986993Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462050948Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462095731Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46222508Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462319676Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462417242Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462457425Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462497098Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462551951Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462617686Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462702312Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462744244Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462799228Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46282877Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462900315Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462952848Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.462999132Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.463081637Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.463120669Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.463153562Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46327409Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.463363756Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.463542478Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.463620213Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.463893451Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.463995788Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464103866Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464158609Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464200972Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464248075Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464294308Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464388234Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464429567Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46446507Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464532294Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464585178Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464629131Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464683864Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46476563Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464842515Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464894719Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46492206Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.464993515Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465047619Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465101732Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465154896Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465186588Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465193428Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465228111Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465396442Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465580154Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465736065Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.465948559Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466147692Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466159163Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-qs6hm/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466202806Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-qs6hm/network-check-source-7d88cb55d8-qs6hm.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466273691Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-qs6hm/check-endpoints/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466280991Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-qs6hm/check-endpoints/check-endpoints/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466285112Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-qs6hm/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466313303Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-qs6hm/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466424861Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-qs6hm/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466500626Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-qs6hm/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466552519Z namespaces/openshift-network-diagnostics/pods/network-check-target-9c8w8/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466586832Z namespaces/openshift-network-diagnostics/pods/network-check-target-9c8w8/network-check-target-9c8w8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466684328Z namespaces/openshift-network-diagnostics/pods/network-check-target-9c8w8/network-check-target-container/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466694949Z namespaces/openshift-network-diagnostics/pods/network-check-target-9c8w8/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466698989Z namespaces/openshift-network-diagnostics/pods/network-check-target-9c8w8/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466733351Z namespaces/openshift-network-diagnostics/pods/network-check-target-9c8w8/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466818477Z namespaces/openshift-network-diagnostics/pods/network-check-target-9c8w8/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466890382Z namespaces/openshift-network-diagnostics/pods/network-check-target-9c8w8/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466922694Z namespaces/openshift-network-diagnostics/pods/network-check-target-gz5d7/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.466959137Z namespaces/openshift-network-diagnostics/pods/network-check-target-gz5d7/network-check-target-gz5d7.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467029431Z namespaces/openshift-network-diagnostics/pods/network-check-target-gz5d7/network-check-target-container/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467036582Z namespaces/openshift-network-diagnostics/pods/network-check-target-gz5d7/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467040562Z namespaces/openshift-network-diagnostics/pods/network-check-target-gz5d7/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467075605Z namespaces/openshift-network-diagnostics/pods/network-check-target-gz5d7/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467184832Z namespaces/openshift-network-diagnostics/pods/network-check-target-gz5d7/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467242426Z namespaces/openshift-network-diagnostics/pods/network-check-target-gz5d7/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467292169Z namespaces/openshift-network-diagnostics/pods/network-check-target-jzkpr/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4673068Z namespaces/openshift-network-diagnostics/pods/network-check-target-jzkpr/network-check-target-jzkpr.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46744665Z namespaces/openshift-network-diagnostics/pods/network-check-target-jzkpr/network-check-target-container/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46745892Z namespaces/openshift-network-diagnostics/pods/network-check-target-jzkpr/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467463531Z namespaces/openshift-network-diagnostics/pods/network-check-target-jzkpr/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467470811Z namespaces/openshift-network-diagnostics/pods/network-check-target-jzkpr/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467517194Z namespaces/openshift-network-diagnostics/pods/network-check-target-jzkpr/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.46759464Z namespaces/openshift-network-diagnostics/pods/network-check-target-jzkpr/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467614431Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467772181Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467817934Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467826125Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467857107Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.467955464Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468007177Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468035479Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468121715Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468169888Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468210541Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468266514Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468316728Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468400493Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468459807Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468523912Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468562464Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468663701Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468746746Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468867545Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468887736Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.468961221Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469005994Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469020485Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469114531Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469154824Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469188206Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469272692Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469320845Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469329186Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469370068Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469415751Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469462285Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469568822Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469661868Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469749404Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469823589Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.469905834Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470056954Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470167462Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470215535Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470255968Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470310881Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470355384Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470402917Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470454541Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470497944Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470543967Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470623112Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470726269Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470772792Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470830016Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470859508Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470902621Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470950004Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470957005Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.470993737Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.471148947Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47132765Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.471472159Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.471694824Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.471888367Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47192523Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.471978423Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.471990864Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472039237Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47208098Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472130153Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472185967Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472238491Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472285083Z namespaces/openshift-network-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472344018Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472392031Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472446574Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472478676Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47252699Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472629687Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472728643Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472805969Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472852892Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472901255Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.472961729Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473024073Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473069716Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473111969Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473205265Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473233247Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47327587Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47357677Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473622063Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473629504Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473681927Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473742061Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.473860939Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474009189Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474091725Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474261076Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474337871Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474525964Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474628921Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474782911Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474848426Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474874867Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.474963683Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475041038Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475089532Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475145075Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475178957Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475235571Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475274504Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475308376Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47536255Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475432345Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47551313Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475546692Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475592895Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475648939Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475860983Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475934138Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.475978781Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.476035815Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47611151Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.476165184Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.476172834Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.476204286Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.476361037Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47655048Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47670957Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.476928075Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47715171Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47715954Z namespaces/openshift-network-operator/pods/iptables-alerter-7pp8b/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477194012Z namespaces/openshift-network-operator/pods/iptables-alerter-7pp8b/iptables-alerter-7pp8b.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477273658Z namespaces/openshift-network-operator/pods/iptables-alerter-7pp8b/iptables-alerter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477282918Z namespaces/openshift-network-operator/pods/iptables-alerter-7pp8b/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477287019Z namespaces/openshift-network-operator/pods/iptables-alerter-7pp8b/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477316801Z namespaces/openshift-network-operator/pods/iptables-alerter-7pp8b/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477405017Z namespaces/openshift-network-operator/pods/iptables-alerter-7pp8b/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477472461Z namespaces/openshift-network-operator/pods/iptables-alerter-7pp8b/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477498123Z namespaces/openshift-network-operator/pods/iptables-alerter-8gs4b/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477543546Z namespaces/openshift-network-operator/pods/iptables-alerter-8gs4b/iptables-alerter-8gs4b.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477616031Z namespaces/openshift-network-operator/pods/iptables-alerter-8gs4b/iptables-alerter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477623011Z namespaces/openshift-network-operator/pods/iptables-alerter-8gs4b/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477627032Z namespaces/openshift-network-operator/pods/iptables-alerter-8gs4b/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477684416Z namespaces/openshift-network-operator/pods/iptables-alerter-8gs4b/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477784742Z namespaces/openshift-network-operator/pods/iptables-alerter-8gs4b/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477852257Z namespaces/openshift-network-operator/pods/iptables-alerter-8gs4b/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477877718Z namespaces/openshift-network-operator/pods/iptables-alerter-np54t/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477924192Z namespaces/openshift-network-operator/pods/iptables-alerter-np54t/iptables-alerter-np54t.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.477996216Z namespaces/openshift-network-operator/pods/iptables-alerter-np54t/iptables-alerter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478003477Z namespaces/openshift-network-operator/pods/iptables-alerter-np54t/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478007377Z namespaces/openshift-network-operator/pods/iptables-alerter-np54t/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478032359Z namespaces/openshift-network-operator/pods/iptables-alerter-np54t/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478132575Z namespaces/openshift-network-operator/pods/iptables-alerter-np54t/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478218321Z namespaces/openshift-network-operator/pods/iptables-alerter-np54t/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478258364Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478293416Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.47835388Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478371711Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478437736Z namespaces/openshift-node/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478448067Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478453687Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478474289Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478636589Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478841743Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.478994113Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.479191726Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.479637196Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.479705941Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.479778886Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.479823249Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.479864892Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.479910475Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480004231Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480094397Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480167542Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480212015Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480253798Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4802939Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480342254Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480419909Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480455021Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480507475Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.48058559Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480630173Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480712138Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.480798554Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.48088861Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481062412Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481072383Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481144697Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481282586Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481469219Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481540504Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481570276Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.48163308Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481698905Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481746738Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481794621Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481878697Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.481954431Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482001545Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482042188Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482086271Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482129113Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482193978Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482223289Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482301445Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482310286Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482358229Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482510669Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482715182Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.482865293Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483052575Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483237498Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.48327647Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483332934Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483372907Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483440741Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483470573Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483521047Z namespaces/openshift-operators/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483530587Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483537128Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483560089Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.483853189Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.484047582Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.484188872Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.484395825Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.484798932Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.484844105Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.484904799Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.484929381Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485028488Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485050929Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485115554Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485138185Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.48535355Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485436425Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485520431Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485552183Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485590705Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485771378Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.48581461Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485850863Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485902316Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.485983452Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486016734Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486051426Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486135992Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486177214Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486184735Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486242329Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486287642Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486323004Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486596383Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.486696469Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.487050203Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.487194263Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.487717898Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.487823795Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488018558Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488092263Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488140046Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488246333Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488392543Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488431246Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488545173Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488605977Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488679202Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488742546Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488804071Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488879896Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488935259Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.488999694Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489006904Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489051987Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489119282Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489169845Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489268942Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489365928Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489427432Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489478616Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.4895412Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489582063Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489706911Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489758455Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489816938Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489823889Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489870432Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.489970939Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.490031963Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.490066825Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.49014158Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.490148801Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.490206955Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.490373326Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.490576199Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.490748911Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.490979256Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.49118917Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491197091Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491273006Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovnkube-node-64lk7.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.49148562Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-node/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491499961Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491505342Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491542784Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491709115Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491793771Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491843814Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491851095Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491855305Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.491892357Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.49207358Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492154885Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492207389Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/nbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492215319Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/nbdb/nbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492219319Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/nbdb/nbdb/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492259872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/nbdb/nbdb/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.49237359Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492455335Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492504929Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/northd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492511869Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/northd/northd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492516029Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/northd/northd/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492564703Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/northd/northd/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492799568Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492887764Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/northd/northd/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492943888Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-acl-logging/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492951428Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492962669Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.492997772Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493112129Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493248238Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493293991Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493301612Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-controller/ovn-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493305592Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493342715Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493713799Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493798855Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493846919Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovnkube-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493854579Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493858639Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.493894742Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.495898596Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.495986402Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496040475Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/sbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496074378Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/sbdb/sbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496082838Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/sbdb/sbdb/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496134622Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/sbdb/sbdb/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496245389Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496328245Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-64lk7/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496379158Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496459804Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovnkube-node-b648k.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496626525Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-node/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496634095Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496656817Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496716191Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496864181Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.496940906Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.49700148Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497009771Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497013771Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497061764Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497212884Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497294199Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497340363Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/nbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497348533Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/nbdb/nbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497352534Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/nbdb/nbdb/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497396146Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/nbdb/nbdb/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497509144Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497590249Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497659294Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/northd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497674305Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/northd/northd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497678225Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/northd/northd/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497722058Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/northd/northd/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497856557Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497938343Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/northd/northd/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.497989686Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-acl-logging/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498014488Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498018728Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498024919Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498140306Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498222462Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498273195Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498281366Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-controller/ovn-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498285446Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498324248Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498669832Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498754347Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.49879474Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovnkube-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498804411Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498808451Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.498858374Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50088132Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.500952795Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501011649Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/sbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501021939Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/sbdb/sbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50102774Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/sbdb/sbdb/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501070423Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/sbdb/sbdb/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50118422Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501272296Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b648k/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501322309Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501405565Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovnkube-node-nrwj7.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501566056Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-node/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501575897Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501581937Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50162138Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501796171Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501880517Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50193227Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501948502Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501955282Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.501975433Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502139074Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.5022253Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502277084Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/nbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502287094Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/nbdb/nbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502293005Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/nbdb/nbdb/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502332267Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/nbdb/nbdb/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502450485Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502535071Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502586964Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/northd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502596685Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/northd/northd/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502602705Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/northd/northd/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502661739Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/northd/northd/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502783567Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502862723Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/northd/northd/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502916506Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-acl-logging/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502926487Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502932607Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.502961999Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503069386Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503154512Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503203575Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503213206Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-controller/ovn-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503219376Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503256569Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503594282Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503691748Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503732041Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovnkube-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503736701Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503742842Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.503789805Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.505762927Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.505845763Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.505892986Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/sbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.505899706Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/sbdb/sbdb/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.505903856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/sbdb/sbdb/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.5059522Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/sbdb/sbdb/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506066938Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506152853Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nrwj7/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506197476Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50625951Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506300353Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506308154Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506498836Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506583972Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506753013Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506810657Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50685106Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.506962927Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507056614Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507115838Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50715547Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507221465Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507233436Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507237516Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507285869Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507351044Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507358044Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507402427Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507606771Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.507858527Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50804837Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.508277026Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.508508791Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.508570405Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.508664992Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.508811921Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.508870525Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.508914558Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509025836Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509164195Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509273902Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509368019Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509411902Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509480196Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.50952807Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509626706Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509716302Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509764025Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509861042Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509920476Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509930377Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.509974439Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.510045794Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.510091897Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.510224506Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.510329123Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.510768373Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.510863129Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511007489Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511155368Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511359032Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511428647Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.51147853Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511545545Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511601298Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511676603Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511743788Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511836934Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.511940721Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512002055Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512042488Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512104902Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512157266Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.51222077Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512266633Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512318506Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512329007Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512382361Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512546872Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512763966Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.512926987Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513161213Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513378937Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513387328Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-fggn6/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513432631Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-fggn6/service-ca-operator-5d6c75f84f-fggn6.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513527117Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-fggn6/service-ca-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513534518Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-fggn6/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513539238Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-fggn6/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513696059Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-fggn6/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.513899962Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-fggn6/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514083215Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514133898Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514195612Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514246286Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514300939Z namespaces/openshift-service-ca/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514356823Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514424107Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514477111Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514530165Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514581798Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514695156Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514819944Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514935022Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.514986855Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515025688Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515090132Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515129605Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515228291Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515287305Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515325948Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515428975Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515477068Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515484929Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515530772Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515596226Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515662681Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.51579543Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.515891606Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.516024325Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.516116581Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.51624601Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.51639297Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.516587023Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.516653307Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.516777365Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.516836189Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.516878732Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.516937966Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517033312Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517114498Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517206084Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517267308Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517324752Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517395877Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.51744675Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517497384Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517505184Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517549617Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.517732039Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518124035Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518292417Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518511001Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518737667Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518753938Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-7kqkc/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518780809Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-7kqkc/service-ca-658d7b7c78-7kqkc.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518872196Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-7kqkc/service-ca-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518880636Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-7kqkc/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518889727Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-7kqkc/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.518930229Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-7kqkc/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519209198Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-7kqkc/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519297434Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-7kqkc/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519327666Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.5193897Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519428183Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519489377Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519547191Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519607705Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519691991Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519731633Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519793987Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.51982967Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.519936047Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520044154Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52013625Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520193174Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520231097Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520307992Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520361445Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520455952Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520515196Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520547948Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520661645Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52073541Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520773613Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.520902962Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521007479Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521110906Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521209302Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521301659Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521453098Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521637281Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521714676Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52177112Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521829424Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521882887Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521932811Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.521984834Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522093092Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522175447Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522239751Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522290025Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522343568Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522392092Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522448355Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522455616Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522501789Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522683761Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.522882914Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523051496Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52326742Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523476734Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523528837Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523571931Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523632104Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523721361Z namespaces/openshift/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523771244Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523836298Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523890482Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.523984708Z namespaces/openshift/apps/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524034911Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524130088Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524229284Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524320201Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524383315Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524433338Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524483372Z namespaces/openshift/batch/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524534935Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524656933Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524708477Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52475551Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524847286Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52490293Z namespaces/openshift/core/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.524963264Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.525092713Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.525186379Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.525282385Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.525376852Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.525471178Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.525624888Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.525907167Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.525968171Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.526006504Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.526067098Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.526118871Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52744713Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.527483283Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.527604211Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.527723479Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.527860358Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.527983136Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.528084963Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.528273835Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.528465648Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.528615229Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.528759778Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.528866695Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.529020505Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.529138964Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.529285593Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.529391121Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.529546551Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52968761Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.52982782Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.529968599Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.530086047Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.530215896Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.530331103Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.530448311Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.530566079Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.530708289Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.530837877Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.530959806Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.531099995Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.531204562Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.531334811Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.53146905Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.531607259Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.531739468Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.531869877Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.531970943Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.532117433Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.532285055Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.532392632Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.532531771Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.532775137Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.532911836Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.533081108Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.533187385Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.533296652Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.533448892Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.533602243Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.533752283Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.533886272Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.533994799Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.534112487Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.534236435Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.534385205Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.534535235Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.534698536Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.534831045Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.534974904Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535088632Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.5352039Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.53534552Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535442716Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.5354978Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535598816Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535715574Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535773418Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535820381Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535876995Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535933649Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535990363Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.535999083Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.536037616Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.536355987Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.536559411Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.536734573Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.536970008Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537195453Z namespaces/openshift/policy/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537239516Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.53729643Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537354634Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537419848Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537426869Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537473842Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537611001Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537780623Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.537915312Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.53804308Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.538251424Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.538378513Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.538521942Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.538662272Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.538826873Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.538957182Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.53909196Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.53923955Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.539408462Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.539541311Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.539708502Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.539836961Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.5399762Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.540107439Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.540232207Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.540362656Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.540565309Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.540806445Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.540934734Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.541068783Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.541183761Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.541305529Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.541428677Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.541569637Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.541721147Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.541893598Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.542026247Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.542143445Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.542291345Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.542426844Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.542544702Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.54281826Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.542943389Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.543087728Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.543228398Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.543355936Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.543496306Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.543631475Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.543807427Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.543955446Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.544105256Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.544275278Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.544430678Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.544570998Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.544725958Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.54491241Z network_logs/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.544958894Z network_logs/cluster_scale [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.545084512Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.545167808Z network_logs/multi-networkpolicy [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.545258664Z network_logs/net-attach-def [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.545363871Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.545540933Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.545662521Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.54819083Z nodes/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.548245054Z nodes/debug [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.548359772Z nodes/ip-10-0-131-29.ec2.internal/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.548424006Z nodes/ip-10-0-131-29.ec2.internal/cpu_affinities.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.548875536Z nodes/ip-10-0-131-29.ec2.internal/dmesg [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.549200838Z nodes/ip-10-0-131-29.ec2.internal/ethtool_channels [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.549299424Z nodes/ip-10-0-131-29.ec2.internal/ethtool_features [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.549403431Z nodes/ip-10-0-131-29.ec2.internal/ip-10-0-131-29.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.549867663Z nodes/ip-10-0-131-29.ec2.internal/irq_affinities.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.550044854Z nodes/ip-10-0-131-29.ec2.internal/lscpu [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.550202885Z nodes/ip-10-0-131-29.ec2.internal/lspci [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.550303022Z nodes/ip-10-0-131-29.ec2.internal/podresources.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.5504313Z nodes/ip-10-0-131-29.ec2.internal/pods_info.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.550602842Z nodes/ip-10-0-131-29.ec2.internal/proc_cmdline [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.550714099Z nodes/ip-10-0-131-29.ec2.internal/sysinfo.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.552491238Z nodes/ip-10-0-131-29.ec2.internal/sysinfo.tgz [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.55266415Z nodes/ip-10-0-133-142.ec2.internal/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.552713603Z nodes/ip-10-0-133-142.ec2.internal/cpu_affinities.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.553030895Z nodes/ip-10-0-133-142.ec2.internal/dmesg [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.553291322Z nodes/ip-10-0-133-142.ec2.internal/ethtool_channels [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.553378358Z nodes/ip-10-0-133-142.ec2.internal/ethtool_features [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.553476615Z nodes/ip-10-0-133-142.ec2.internal/ip-10-0-133-142.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.553920494Z nodes/ip-10-0-133-142.ec2.internal/irq_affinities.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.554088645Z nodes/ip-10-0-133-142.ec2.internal/lscpu [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.554241516Z nodes/ip-10-0-133-142.ec2.internal/lspci [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.554347503Z nodes/ip-10-0-133-142.ec2.internal/podresources.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.554475951Z nodes/ip-10-0-133-142.ec2.internal/pods_info.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.55460731Z nodes/ip-10-0-133-142.ec2.internal/proc_cmdline [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.554735249Z nodes/ip-10-0-133-142.ec2.internal/sysinfo.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.556520638Z nodes/ip-10-0-133-142.ec2.internal/sysinfo.tgz [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.556790586Z nodes/ip-10-0-135-177.ec2.internal/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.55683829Z nodes/ip-10-0-135-177.ec2.internal/cpu_affinities.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.557161771Z nodes/ip-10-0-135-177.ec2.internal/dmesg [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.557483803Z nodes/ip-10-0-135-177.ec2.internal/ethtool_channels [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.557565458Z nodes/ip-10-0-135-177.ec2.internal/ethtool_features [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.557687287Z nodes/ip-10-0-135-177.ec2.internal/ip-10-0-135-177.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.558203771Z nodes/ip-10-0-135-177.ec2.internal/irq_affinities.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.558370323Z nodes/ip-10-0-135-177.ec2.internal/lscpu [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.558522363Z nodes/ip-10-0-135-177.ec2.internal/lspci [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.558624429Z nodes/ip-10-0-135-177.ec2.internal/podresources.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.558771119Z nodes/ip-10-0-135-177.ec2.internal/pods_info.json [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.558906198Z nodes/ip-10-0-135-177.ec2.internal/proc_cmdline [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.559002805Z nodes/ip-10-0-135-177.ec2.internal/sysinfo.log [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.560918563Z nodes/ip-10-0-135-177.ec2.internal/sysinfo.tgz [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.561106946Z pod_network_connectivity_check/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.561180721Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.561292758Z static-pods/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.561300199Z static-pods/kube-apiserver/ [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.569782917Z [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.569799938Z sent 72,246 bytes received 8,413,803 bytes 5,657,366.00 bytes/sec [must-gather] [must-gather-tn7p8] OUT 2026-07-28T19:07:18.569804339Z total size is 106,381,166 speedup is 12.54 [must-gather] [must-gather ] OUT 2026-07-28T19:07:18.893101093Z namespace/openshift-must-gather-rd9v9 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 3ff16daa-3fce-4b00-be0a-650b5c215b4a [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-jfrjl [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-jfrjl-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-jfrjl [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 16602246 Jul 28 19:09 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-jfrjl/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts f3a8c34] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-jfrjl/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 1869938..f3a8c34 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-28T19:09:05.657Z","type":3}]