task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-l7ktb-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-l7ktb-kubeconfig [get-kubeconfig] Found admin password secret: cluster-l7ktb-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-l7ktb-password [get-kubeconfig] API Server URL: https://a0efbcaf2b82142f4a016b251d320527-bc087ef89d600dae.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.285f140b-bfbb-4f11-b960-0b299617d7ab.prod.konfluxeaas.com [clone-repo] auto-sync/master-to-release-v0.17 [clone-repo] https://github.com/opendatahub-io/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 46% (1509/3261) Updating files: 47% (1533/3261) Updating files: 48% (1566/3261) Updating files: 49% (1598/3261) Updating files: 50% (1631/3261) Updating files: 51% (1664/3261) Updating files: 52% (1696/3261) Updating files: 53% (1729/3261) Updating files: 54% (1761/3261) Updating files: 55% (1794/3261) Updating files: 56% (1827/3261) Updating files: 57% (1859/3261) Updating files: 58% (1892/3261) Updating files: 59% (1924/3261) Updating files: 60% (1957/3261) Updating files: 61% (1990/3261) Updating files: 62% (2022/3261) Updating files: 63% (2055/3261) Updating files: 64% (2088/3261) Updating files: 65% (2120/3261) Updating files: 66% (2153/3261) Updating files: 67% (2185/3261) Updating files: 68% (2218/3261) Updating files: 69% (2251/3261) Updating files: 70% (2283/3261) Updating files: 71% (2316/3261) Updating files: 72% (2348/3261) Updating files: 73% (2381/3261) Updating files: 74% (2414/3261) Updating files: 75% (2446/3261) Updating files: 76% (2479/3261) Updating files: 77% (2511/3261) Updating files: 78% (2544/3261) Updating files: 79% (2577/3261) Updating files: 80% (2609/3261) Updating files: 81% (2642/3261) Updating files: 82% (2675/3261) Updating files: 83% (2707/3261) Updating files: 84% (2740/3261) Updating files: 85% (2772/3261) Updating files: 86% (2805/3261) Updating files: 87% (2838/3261) Updating files: 88% (2870/3261) Updating files: 89% (2903/3261) Updating files: 90% (2935/3261) Updating files: 91% (2968/3261) Updating files: 92% (3001/3261) Updating files: 93% (3033/3261) Updating files: 94% (3066/3261) Updating files: 95% (3098/3261) Updating files: 96% (3131/3261) Updating files: 97% (3164/3261) Updating files: 98% (3196/3261) Updating files: 99% (3229/3261) Updating files: 100% (3261/3261) Updating files: 100% (3261/3261), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1814 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1814 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Prepared 101 packages in 1.87s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 464ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 49ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a0efbcaf2b82142f4a016b251d320527-bc087ef89d600dae.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-tnpgh 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-tnpgh condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-tgrjg 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-tgrjg condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-7qbjb 1/1 Running 0 51s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-7qbjb condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-stable [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-stable [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-7584bddd84-vvhxz 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-7584bddd84-vvhxz condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-zv4dc 1/1 Running 0 11s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-zv4dc condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-8bfqq 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-8bfqq condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-8699b59bbc-spfsb 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-8699b59bbc-spfsb condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-t4rz2 1/1 Running 0 67s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-t4rz2 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-t4rz2 1/1 Running 0 77s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-t4rz2 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-l7ktb-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-l7ktb-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-l7ktb-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-l7ktb-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-l7ktb-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-l7ktb-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Mon Jul 27 17:41:50 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-z4ltk condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-ff96d5959-qz9gv" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-ff96d5959-nxnm8 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:62ab60f9c45b7c3b07bf30293f6be92e64bdd759fbf36392332635abe05f9e67 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:b5c835d640b650f164ba7f36d7008760c88e2974efa65853ee167323c441580c [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'cov': '5.0.0', 'asyncio': '0.23.8', 'anyio': '4.9.0', 'httpx': '0.30.0', 'metadata': '3.1.1', 'xdist': '3.6.1', 'json-report': '1.5.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: cov-5.0.0, asyncio-0.23.8, anyio-4.9.0, httpx-0.30.0, metadata-3.1.1, xdist-3.6.1, json-report-1.5.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] 2026-07-27 17:44:21.888 6283 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-27 17:44:21.888 6280 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-27 17:46:55.869 6280 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-27 17:46:55.869 6280 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-27 17:46:55.899 6280 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:46:55.899 6280 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:46:55.909 6280 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:46:55.909 6280 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-27 17:47:03.767 6280 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-27 17:47:03.767 6280 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-27 17:47:03.794 6280 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:47:03.794 6280 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:47:03.800 6280 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:47:03.800 6280 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T17:56:29.161286', start_time = 1785174989.1616926 [e2e-llm-inference-service] duration = 900.2254230976105, timestamp_end = '2026-07-27T18:11:29.387143' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f1882c042c0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-27T17:56:29.089780] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T17:56:29.105539] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T17:56:29.161048] end - ✅ in 0.055s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T17:56:29.161286] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:56:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 3: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 4: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 5: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:11:29.387143] end - ❌ 900.225s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T18:11:29.387353] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-vvj8m'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T18:11:49.505811] end - ✅ in 20.118s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-27T18:11:49.505981] end - ❌ 920.416s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:56:49Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_deployment(test_case: TestCase): [e2e-llm-inference-service] """KEDA + Deployment: ScaledObject exists with WVA annotations; no HPA; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:606: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T18:11:50.283391', start_time = 1785175910.2836723 [e2e-llm-inference-service] duration = 900.3077261447906, timestamp_end = '2026-07-27T18:26:50.591401' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f1882265760> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_deployment] [2026-07-27T18:11:50.096517] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T18:11:50.109096] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T18:11:50.283245] end - ✅ in 0.174s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T18:11:50.283391] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:12:02Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'message': 'ScaledObject conditions not yet available', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-keda-deploy-inference-pool-ip-d28561d2.e2e-test-llm-autoscaling-keda-deployment-b2150d0b.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-keda-deploy-inference-pool-ip-d28561d2.e2e-test-llm-autoscaling-keda-deployment-b2150d0b.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "BackendNotFound", message "backend(autoscale-keda-deploy-inference-pool-ip-d28561d2.e2e-test-llm-autoscaling-keda-deployment-b2150d0b.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 6: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 8: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:26:50.591401] end - ❌ 900.308s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T18:26:50.591484] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-ndght'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-77fc6d7cf9-4s67l', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-ndght'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-ndght'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-ndght'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T18:27:10.700999] end - ✅ in 20.109s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_deployment] [2026-07-27T18:27:10.701125] end - ❌ 920.604s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:02Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:12:33Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:12:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_lws(test_case: TestCase): [e2e-llm-inference-service] """HPA + LWS: HPA exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:670: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...ale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T18:27:12.139360', start_time = 1785176832.1396413 [e2e-llm-inference-service] duration = 900.8573551177979, timestamp_end = '2026-07-27T18:42:12.996998' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f18822653a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_lws] [2026-07-27T18:27:11.574665] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T18:27:11.587353] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T18:27:12.139226] end - ✅ in 0.552s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T18:27:12.139360] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:27:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 10: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:42:12.996998] end - ❌ 900.857s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T18:42:12.997085] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-bk727'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-bk727'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-bk727'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-bk727'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T18:42:38.249613] end - ✅ in 25.252s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_lws] [2026-07-27T18:42:38.249750] end - ❌ 926.675s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:27:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:27:53Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:27:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T18:32:27.144264', start_time = 1785177147.144522 [e2e-llm-inference-service] duration = 900.7281386852264, timestamp_end = '2026-07-27T18:47:27.872663' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fb9ea75da80> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:34:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-48639af5 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-27T18:32:27.003512] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-48639af5', before_test=[ at 0x7fb9eaf9afc0>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T18:32:27.015955] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T18:32:27.144103] end - ✅ in 0.128s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T18:32:27.144264] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'message': 'Inference Pool e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:34:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:34:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:47:27.872663] end - ❌ 900.728s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:34:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:34:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'e2e-test-llm-inference-service-48639af5' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:27Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:27Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:27Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T18:34:51Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] resourceVersion: '70029' [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a352bf1ffc5df49f08c366c1ddd9746c-711202132.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a352bf1ffc5df49f08c366c1ddd9746c-711202132.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a352bf1ffc5df49f08c366c1ddd9746c-711202132.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:47Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:47Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:34:51Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:47Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:38Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:47Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:33:09Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:33:09Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:47Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a352bf1ffc5df49f08c366c1ddd9746c-711202132.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd to ip-10-0-131-114.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:12 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-47e956c4-9962-4d79-8479-81e5daeb1dba" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:14 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:14 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:14 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:14 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:06 e2e-test-llm-inference-service-48639af5 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-m4fpd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:25 e2e-test-llm-inference-service-48639af5 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:06 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:06 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:06 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-ff8df69bd-l2k52_161b2e73-eef6-4895-8135-21e800b7ce7a Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-48639af5/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:09 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-ff8df69bd-l2k52_161b2e73-eef6-4895-8135-21e800b7ce7a Normal ProvisioningSucceeded Successfully provisioned volume pvc-47e956c4-9962-4d79-8479-81e5daeb1dba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm to ip-10-0-131-114.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:44 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-47e956c4-9962-4d79-8479-81e5daeb1dba" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:45 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.132.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:45 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:46 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" in 1.642s (1.642s including waiting). Image size: 166803393 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:46 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:46 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:47 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:47 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:47 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:34:35 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-114.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.55:8001/health": dial tcp 10.132.0.55:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6 to ip-10-0-131-74.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:44 e2e-test-llm-inference-service-48639af5 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-47e956c4-9962-4d79-8479-81e5daeb1dba" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:32 e2e-test-llm-inference-service-48639af5 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-132820f03254470cd8a0146008974a5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:46 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:42:39 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:45 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:36 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz to ip-10-0-131-74.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:37 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.133.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-74.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-74.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 18:32:37 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-131-74.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785177166.9044182,"msg":"Proxy starting","Built on":"","From Git SHA":"unknown"} [e2e-llm-inference-service] {"level":"info","ts":1785177166.9044414,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"e2e-test-llm-inference-service-48639af5\",\"InferencePoolName\":\"llmisvc-model-pvc-router-manage-e8706282-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1785177166.9050674,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"e2e-test-llm-inference-service-48639af5","poolName":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1785177166.9185739,"logger":"allowlist-validator","msg":"InferencePool added","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177166.9274318,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785177166.9274864,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785177167.0053596,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785177167.0063632,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1785177167.0063748,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1785177167.641909,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177168.6460211,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177185.0830028,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177196.9201202,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177196.929863,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177196.92991,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177226.9215703,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177226.9323108,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177226.9323597,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177256.9234123,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177256.9327152,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177256.9327872,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177285.0838196,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177285.0981126,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177286.9237077,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177286.933444,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177286.933515,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177316.9248037,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177316.935946,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177316.9360228,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177346.9254022,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177346.9361818,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177346.9362288,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177376.9257352,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177376.9353445,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177376.9353964,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177406.9261189,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177406.9386294,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177406.9386835,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177436.9270496,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177436.9368722,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177436.9369218,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177466.9273112,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177466.9374077,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177466.9374652,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177496.9276466,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177496.9373307,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177496.937381,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177526.928213,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177526.9393206,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177526.9393744,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177556.929254,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177556.9391227,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177556.939171,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177586.9299204,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177586.94064,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177586.940699,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177616.9306421,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177616.9409063,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177616.9409666,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177646.930833,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177646.9421546,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177646.9422283,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177676.9309156,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177676.9407146,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177676.9407783,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177706.9311733,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177706.9411707,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177706.9412603,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177736.9318404,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177736.941664,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177736.9417205,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177766.9325392,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177766.942241,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177766.9422941,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177796.9334147,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177796.9439118,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177796.943964,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177826.9334872,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177826.9432569,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177826.943316,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177856.9342637,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177856.9439042,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177856.9439561,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177886.9348843,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177886.9438589,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177886.9439175,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177916.9355862,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177916.9455438,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177916.9455945,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177946.9366677,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177946.9458027,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177946.9458637,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177976.93747,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785177976.9473922,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177976.9474587,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785178006.9384212,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785178006.9471874,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785178006.9472415,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785178036.9384968,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785178036.9479475,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785178036.9480028,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.55":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm":{}}} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 (EngineCore pid=74) INFO 07-27 18:33:57 [v1/worker/cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W727 18:33:57.086937511 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W727 18:33:57.086969549 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 74: [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [v1/worker/cpu_worker.py:109] OMP tid: 74, core 4 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [v1/worker/cpu_worker.py:109] OMP tid: 91, core 5 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [v1/worker/cpu_worker.py:109] OMP tid: 92, core 6 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [v1/worker/cpu_worker.py:109] OMP tid: 93, core 7 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [v1/worker/cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:33:57 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.55:46659 backend=gloo [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.55:46659 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:33:57 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:33:57 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors). [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:33:57 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading). [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:33:57 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:33:57 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:33:57 [config/compilation.py:1194] enabled custom ops: Counter() [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:33:57 [config/compilation.py:1195] disabled custom ops: Counter({'vocab_parallel_embedding': 1, 'logits_processor': 1}) [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:33:57 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ... [e2e-llm-inference-service] (EngineCore pid=74) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00 [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:07 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:17 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:27 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:34:34 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/86c9c3c579382eef68a98ac1d59b39811ba08abef3b4e90675a32c8dec3d7c90/rank_0_0/model [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:34:36 [compilation/monitor.py:76] Initial profiling/warmup run took 1.04 s [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:34:36 [v1/worker/cpu_model_runner.py:92] Warming up done. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:34:36 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 37.26 seconds [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:34:36 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode='hf' [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:34:36 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 18:34:36 [config/vllm.py:790] Asynchronous scheduling is disabled. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:36 [v1/engine/utils.py:1158] READY from local core engine process 0. [e2e-llm-inference-service] (EngineCore pid=74) WARNING 07-27 18:34:36 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:34:36 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:36 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:34:36 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 18:34:36 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:36 [entrypoints/openai/api_server.py:590] Supported tasks: ['generate'] [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/base.py:197] Warming up chat template processing... [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:141] There is no chat template fallback for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [renderers/hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:37 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /docs, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /redoc, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:37 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:45 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:45 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:45 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:45 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:45 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:45 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:45 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 18:34:45 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:34:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:35:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:35:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:35:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:35:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:35:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:35:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:36:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:36:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:36:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:36:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:36:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:36:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:37:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:37:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:37:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:37:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:37:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:37:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:38:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:38:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:38:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:38:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:38:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:38:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:39:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:39:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:39:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:39:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:39:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:39:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:40:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:40:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:40:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:40:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:40:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:40:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:41:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:41:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:41:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:41:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:41:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:41:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:42:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:42:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:42:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:42:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:42:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:42:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:43:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:43:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:43:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:43:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:43:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:43:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:44:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:44:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:44:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:44:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:44:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:44:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:45:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:45:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:45:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:45:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:45:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:45:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:46:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:46:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:46:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:46:37 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:46:47 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:46:57 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:47:07 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:47:17 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:47:27 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '6dbe7324-126e-46a5-a917-11e487fbe4c4', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 27 Jul 2026 18:47:28 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785177157.1287394,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1289117,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"http","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"e2e-test-llm-inference-service-48639af5","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1294286,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1305137,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1305528,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1308622,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1496842,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.149761,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1498227,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1498644,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1499064,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1499784,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.15,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1500716,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1501517,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1502082,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1502328,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1502485,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.150258,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.150391,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1504016,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000a0af40 DataConfig:{Sources:[{Plugin:0xc0004bf0e0 Extractors:[0xc000a0b140]}]} FlowControlConfig: ParserRegistry:0xc000a0b800}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.166559,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1665869,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.166605,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1666102,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1666145,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1666193,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1666236,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.166628,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.166632,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.166637,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1666412,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1666455,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.166652,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1666563,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1666958,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1667037,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1667168,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785177157.166744,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1667638,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1668458,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1668625,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1668916,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1670802,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.167332,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785177157.167358,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1675937,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.167598,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1676195,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1677217,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.16818,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1682613,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1720786,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1729193,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1731153,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.1737595,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.2692585,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.2692866,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785177157.2692947,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.269258,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.2693326,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785177157.2693076,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785177157.2694778,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"cad4aa26-39ca-4f43-8624-7df16d6d9739"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.3694596,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785177157.369492,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785177165.8576598,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"9fe6d944-801b-45b9-9405-8a7ab6c885b3"} [e2e-llm-inference-service] {"level":"info","ts":1785177285.0962124,"caller":"inflightload/producer.go:308","msg":"Injected dynamic attribute into endpoint","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm-rank-0"} [e2e-llm-inference-service] {"level":"info","ts":1785177285.0962727,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm","reconcileID":"70c6cf5a-b739-42bb-9066-d7a8eef238ed"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 0ba24e3d-e6e9-4ebb-8742-674bc6ca7608 [e2e-llm-inference-service] resourceVersion: '68970' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.62 [e2e-llm-inference-service] nodeName: ip-10-0-131-74.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz [e2e-llm-inference-service] uid: 4f32425c-6da8-436b-8640-ace532cc8c8d [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 0beed3d7-e00f-405c-a342-b4fdfcb657a5 [e2e-llm-inference-service] resourceVersion: '69965' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.55 [e2e-llm-inference-service] nodeName: ip-10-0-131-114.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm [e2e-llm-inference-service] uid: a8f61f6b-b525-4658-ba38-b97f8631fe50 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a8f61f6b-b525-4658-ba38-b97f8631fe50 [e2e-llm-inference-service] resourceVersion: '69963' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.55/23"],"mac_address":"0a:58:0a:84:00:37","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.55/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.55\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:37\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] uid: a0bbf88b-d2ca-4811-b5fd-6b177b12ad92 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-114 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a0bbf88b-d2ca-4811-b5fd-6b177b12ad92"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.55"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-2qjpd [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-2qjpd [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000980000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-2qjpd [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000980000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-131-114.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c25 [e2e-llm-inference-service] fsGroup: 1000980000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-9xfm7 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-cb9h8 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:47Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:47Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] hostIP: 10.0.131.114 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.114 [e2e-llm-inference-service] podIP: 10.132.0.55 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.55 [e2e-llm-inference-service] startTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:4d57f8fe65a63b44f345487af2b831e64ddf7fb01126d4f51cc8cf49b7412d00 [e2e-llm-inference-service] containerID: cri-o://fc63be0dcb1f24bf4d01b2adc3d10611db243c729993c6cd207c2ad63e1750c0 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-2qjpd [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000980000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000980000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-27T18:32:47Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://b8dc52b204b52518cad26ac19412802f747e01891ffe3752a484e261aea13eff [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-2qjpd [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000980000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000980000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b8lrz6 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: fd309d91-1411-4b4b-bedb-3ae7c53a796b [e2e-llm-inference-service] resourceVersion: '68392' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.61/23"],"mac_address":"0a:58:0a:85:00:3d","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.61/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] uid: 9c8eb7fc-254c-43aa-95aa-1c1ac8465116 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-74 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9c8eb7fc-254c-43aa-95aa-1c1ac8465116"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-hcb72 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-hcb72 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000980000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-131-74.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c25 [e2e-llm-inference-service] fsGroup: 1000980000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-9xfm7 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] hostIP: 10.0.131.74 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.74 [e2e-llm-inference-service] startTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-hcb72 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4f32425c-6da8-436b-8640-ace532cc8c8d [e2e-llm-inference-service] resourceVersion: '68968' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.62/23"],"mac_address":"0a:58:0a:85:00:3e","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.62/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.62\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:3e\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] uid: 5c3c84f4-3dd7-4c4b-ba8a-e26ae79dfb6d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-74 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c3c84f4-3dd7-4c4b-ba8a-e26ae79dfb6d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.62"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-869b6 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef:\ [e2e-llm-inference-service] \ decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-869b6 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000980000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-131-74.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c25 [e2e-llm-inference-service] fsGroup: 1000980000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-9xfm7 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ldrms [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] hostIP: 10.0.131.74 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.74 [e2e-llm-inference-service] podIP: 10.133.0.62 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.62 [e2e-llm-inference-service] startTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://fd0147235c3cc7112d7368b3a9101cf4e2d453946e463d8048d8fb94e9360c4a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-869b6 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000980000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000980000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c3c7bb29-d057-4465-a1e3-cf60e113226f [e2e-llm-inference-service] resourceVersion: '68383' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ldrms [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ldrms"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-9xfm7"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-9xfm7 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ldrms [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-9xfm7 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ldrms [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 774daec5-900b-404c-9b94-3adf4a25eaa8 [e2e-llm-inference-service] resourceVersion: '68343' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-cb9h8 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-cb9h8"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-9xfm7"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-9xfm7 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-cb9h8 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-9xfm7 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-cb9h8 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: bbdd7f4e-41a6-4e32-b22a-57a99265bc61 [e2e-llm-inference-service] resourceVersion: '68405' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.81.249 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.81.249 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: bc67a835-03bd-48dc-9e92-f39eef41363f [e2e-llm-inference-service] resourceVersion: '68370' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.82.139 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.82.139 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d424bba1-cd56-482c-9ab9-2d4f15d29554 [e2e-llm-inference-service] resourceVersion: '69970' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 1b5ba445-17a7-4404-973d-aa5d6bc6738a [e2e-llm-inference-service] resourceVersion: '75235' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:42:37Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T18:42:37Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:42:37Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9a70c1cb-49a2-4a5d-8447-73e17797860d [e2e-llm-inference-service] resourceVersion: '68972' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a0bbf88b-d2ca-4811-b5fd-6b177b12ad92 [e2e-llm-inference-service] resourceVersion: '69968' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: d424bba1-cd56-482c-9ab9-2d4f15d29554 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d424bba1-cd56-482c-9ab9-2d4f15d29554"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9c8eb7fc-254c-43aa-95aa-1c1ac8465116 [e2e-llm-inference-service] resourceVersion: '68388' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: 1b5ba445-17a7-4404-973d-aa5d6bc6738a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1b5ba445-17a7-4404-973d-aa5d6bc6738a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 5c3c84f4-3dd7-4c4b-ba8a-e26ae79dfb6d [e2e-llm-inference-service] resourceVersion: '68971' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: 9a70c1cb-49a2-4a5d-8447-73e17797860d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9a70c1cb-49a2-4a5d-8447-73e17797860d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e8479faf-91f9-453e-8951-98140ea49901 [e2e-llm-inference-service] resourceVersion: '68397' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9e02d2dc-6d10-4471-969f-d2eb638cf9b0 [e2e-llm-inference-service] resourceVersion: '68349' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 0222a55b-02d6-4d19-8f7a-31fa470f0dd6 [e2e-llm-inference-service] resourceVersion: '68395' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2561e85c-7ab9-44c0-90bf-51c8d622fd9d [e2e-llm-inference-service] resourceVersion: '68346' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-hwmw6 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 43e2622c-e431-40ed-825f-9ef0b09c0336 [e2e-llm-inference-service] resourceVersion: '68969' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: bbdd7f4e-41a6-4e32-b22a-57a99265bc61 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:33:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"bbdd7f4e-41a6-4e32-b22a-57a99265bc61"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.62 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz [e2e-llm-inference-service] uid: 4f32425c-6da8-436b-8640-ace532cc8c8d [e2e-llm-inference-service] nodeName: ip-10-0-131-74.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-s5k6bm [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 0cc8ea34-cae3-43b0-83a7-a98b3a0f95bb [e2e-llm-inference-service] resourceVersion: '69967' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: bc67a835-03bd-48dc-9e92-f39eef41363f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:34:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"bc67a835-03bd-48dc-9e92-f39eef41363f"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.55 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm [e2e-llm-inference-service] uid: a8f61f6b-b525-4658-ba38-b97f8631fe50 [e2e-llm-inference-service] nodeName: ip-10-0-131-114.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e8479faf-91f9-453e-8951-98140ea49901 [e2e-llm-inference-service] resourceVersion: '68397' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 9e02d2dc-6d10-4471-969f-d2eb638cf9b0 [e2e-llm-inference-service] resourceVersion: '68349' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 0222a55b-02d6-4d19-8f7a-31fa470f0dd6 [e2e-llm-inference-service] resourceVersion: '68395' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2561e85c-7ab9-44c0-90bf-51c8d622fd9d [e2e-llm-inference-service] resourceVersion: '68346' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68677' [e2e-llm-inference-service] uid: b0ee3a72-d68f-4aef-a8b9-a097b23bf080 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:38Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68677' [e2e-llm-inference-service] uid: b0ee3a72-d68f-4aef-a8b9-a097b23bf080 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:38Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68662' [e2e-llm-inference-service] uid: 0fb26959-a6e0-4fff-aeff-21778479db44 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:45Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T18:32:39Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68582' [e2e-llm-inference-service] uid: 64877b56-105c-4cf4-ab72-4c88ca4936b2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:38Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T18:32:39Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68445' [e2e-llm-inference-service] uid: c513ed64-c4f8-4410-9802-8de9165e31af [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68679' [e2e-llm-inference-service] uid: d741530e-ec0b-46a6-8f87-a6cc8cd68984 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68452' [e2e-llm-inference-service] uid: 250c1cc8-cdca-4b57-8430-01dba1fe34c6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68445' [e2e-llm-inference-service] uid: c513ed64-c4f8-4410-9802-8de9165e31af [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68679' [e2e-llm-inference-service] uid: d741530e-ec0b-46a6-8f87-a6cc8cd68984 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68452' [e2e-llm-inference-service] uid: 250c1cc8-cdca-4b57-8430-01dba1fe34c6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68445' [e2e-llm-inference-service] uid: c513ed64-c4f8-4410-9802-8de9165e31af [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:46Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68679' [e2e-llm-inference-service] uid: d741530e-ec0b-46a6-8f87-a6cc8cd68984 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68452' [e2e-llm-inference-service] uid: 250c1cc8-cdca-4b57-8430-01dba1fe34c6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4bb4e9da-58b2-4418-908b-5ee3921bd322"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T18:32:36Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 4bb4e9da-58b2-4418-908b-5ee3921bd322 [e2e-llm-inference-service] resourceVersion: '68414' [e2e-llm-inference-service] uid: 16b1e191-a4ce-41b7-b263-d05ae5cc0865 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:47:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] timestamp: '2026-07-27T18:47:15Z' [e2e-llm-inference-service] window: 10.066s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 15160758n [e2e-llm-inference-service] memory: 26280Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 97481323n [e2e-llm-inference-service] memory: 2402848Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schep44lz [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:47:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] timestamp: '2026-07-27T18:47:11Z' [e2e-llm-inference-service] window: 17.124s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 56436463n [e2e-llm-inference-service] memory: 33536Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-27T18:47:29.333945] end - ❌ 902.330s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:34:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:33:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:32:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-48639af5 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_lws(test_case: TestCase): [e2e-llm-inference-service] """KEDA + LWS: ScaledObject exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:728: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T18:42:39.647399', start_time = 1785177759.6477528 [e2e-llm-inference-service] duration = 900.2875382900238, timestamp_end = '2026-07-27T18:57:39.935295' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f1882266020> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_lws] [2026-07-27T18:42:39.472335] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T18:42:39.484690] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T18:42:39.647236] end - ✅ in 0.162s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T18:42:39.647399] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:42:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'message': 'ScaledObject conditions not yet available', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 12: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 13: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 14: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:57:39.935295] end - ❌ 900.288s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T18:57:39.935454] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t4228'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t4228'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t4228'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-t4228'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T18:58:10.261780] end - ✅ in 30.326s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_lws] [2026-07-27T18:58:10.261899] end - ❌ 930.789s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:42:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:43:28Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:43:08Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-lws-e541a132 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_cleanup_hpa[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-cleanup-hpa", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_cleanup_hpa(test_case: TestCase): [e2e-llm-inference-service] """Removing scaling config should delete HPA.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:892: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T18:58:11.545117', start_time = 1785178691.545469 [e2e-llm-inference-service] duration = 900.9540784358978, timestamp_end = '2026-07-27T19:13:12.499550' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f18822665c0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_cleanup_hpa] [2026-07-27T18:58:11.381516] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-cleanup-hpa', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T18:58:11.394421] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T18:58:11.544990] end - ✅ in 0.150s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T18:58:11.545117] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:58:34Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 15: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 16: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 17: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:44:40Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T19:13:12.499550] end - ❌ 900.954s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T19:13:12.499720] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-wnpkl', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-rc29b'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-wnpkl', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-rc29b'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-rc29b'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-rc29b'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T19:13:48.144039] end - ✅ in 35.644s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_cleanup_hpa] [2026-07-27T19:13:48.144196] end - ❌ 936.762s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:34Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:59:32Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:58:54Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:92: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 6 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 6 failed, 43 passed, 3 skipped, 25 warnings in 5371.77s (1:29:31) ======= [must-gather] [must-gather ] OUT 2026-07-27T19:13:51.879770475Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 328368b6-397a-4aea-a1e7-1353e5a2e7f8 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-27T19:13:51.920935066Z namespace/openshift-must-gather-sth4s created [must-gather] [must-gather ] OUT 2026-07-27T19:13:51.929765416Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-xq7jk created [must-gather] [must-gather ] OUT 2026-07-27T19:13:51.958967002Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:58.652947184Z [disk usage checker] Started [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:58.656527856Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:58.921118420Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.084973959Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.085004551Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.085004551Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.087412141Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.250991319Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.251022108Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.251022108Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.253043610Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.417967049Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.417997302Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.417997302Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.420156945Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.580842653Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.580872859Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.580872859Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.582898247Z Error getting logs from rhods-notebooks [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.741720631Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.741751781Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.741751781Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.743734067Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.908584702Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.908614159Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.908614159Z namespaces "istio-system" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:13:59.911054516Z Error getting logs from istio-system [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.075323020Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.075353835Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.075353835Z namespaces "knative-serving" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.077751474Z Error getting logs from knative-serving [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.238056876Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.238085725Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.238085725Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.240244614Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.515366015Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.796903639Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:00.932041037Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.018675971Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.020870244Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.148365104Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.231259293Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.233179955Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.360935147Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.445084809Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.447062921Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.579060110Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.661788103Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.663694431Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.799284071Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.881838797Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:01.883932195Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.016320896Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.105130594Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.107278529Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.240583213Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.325297344Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.327555659Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.459745802Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.544294476Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.546403598Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.679280087Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.763452539Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.765716223Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.896383238Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.979034665Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:02.981224899Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.114701395Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.195239560Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.197068458Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.330097516Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.415478214Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.417722669Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.545025277Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.628795472Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.630870346Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.661147397Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.764354782Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.847276951Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.849408149Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:03.980269351Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.066334001Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.068615507Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.203570784Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.289139060Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.291584273Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.419542248Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.505013247Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.507192541Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.634913188Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.717952978Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:04.720008264Z Error collecting info from [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:05.400795221Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:05.529318544Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:05.656551649Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:05.787984831Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:05.914672770Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:06.492747260Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:06.616684518Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:06.746658880Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:06.877745323Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:07.006722642Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:07.533894883Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:07.677323034Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:07.759936229Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:07.829514824Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:07.986214522Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.114876456Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.201525790Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.276270791Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.434256338Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.557189956Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.626291562Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.668070874Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.700971942Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.859391923Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:08.992145138Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.061470943Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.129960856Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.290012976Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.421402577Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.496273967Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.566943145Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.725725115Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.856114892Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.925750949Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:09.994724865Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:10.149553080Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:10.270827815Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:10.342719230Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:10.411103615Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:10.572788622Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:10.698329279Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:10.771976339Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:10.844679932Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.008068277Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.136342582Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.211860891Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.278731582Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.438156002Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.568063072Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.638528056Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.706657161Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.866599654Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:11.995757811Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.068678559Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.139734583Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.297661775Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.427849509Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.507471764Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.581263120Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.741240132Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.867599680Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:12.958262903Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:13.030777004Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:13.187311834Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:13.318309437Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:13.390357833Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:13.456476680Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:13.620270904Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:13.673615082Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:14.292611537Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:14.437955860Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:14.546530788Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:14.708057630Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:15.359266636Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:15.521842441Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:15.629558652Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:15.794653622Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:16.469073872Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:16.620286342Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:16.726345110Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:16.890629322Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:17.701352414Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:17.876671264Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:17.994909567Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:18.164416513Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:18.678884550Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:18.999821651Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:19.188222533Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:19.309066908Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:19.473096365Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:20.356416617Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:20.537522698Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:20.661755455Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:20.825403922Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:21.669804202Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:21.845924915Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:21.968645440Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:22.132746286Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:22.953252452Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:23.128383893Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:23.249492996Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:23.416334643Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:23.683586190Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:24.255861137Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:24.433764299Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:24.556849556Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:24.713055211Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:25.557255051Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:25.731147348Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:25.852870925Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:26.010585221Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:26.875019268Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:27.056914471Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:27.173297127Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:27.330234280Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:28.136122111Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:28.305785954Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:28.422104744Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:28.584938702Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:28.688879169Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:29.424484715Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:29.612772443Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:29.733767667Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:29.895386376Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:30.734622588Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:30.913032726Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:31.039743541Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:31.199123411Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:32.022355200Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:32.200986978Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:32.322590057Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:32.487192965Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:33.330809154Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:33.509664340Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:33.628253888Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:33.694703136Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:33.792845416Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:34.638248535Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:34.818809817Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:34.942198925Z Wrote inspect data to must-gather. [must-gather] [must-gather-k76f6] POD 2026-07-27T19:14:34.962858728Z Caches written to disk [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:35.971077363Z waiting for gather to complete [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:35.974151205Z downloading gather output [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.259010852Z receiving incremental file list [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.27222714Z ./ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.272320771Z aggregated-discovery-api.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.272650128Z aggregated-discovery-apis.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.273550936Z event-filter.html [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.273944544Z timestamp [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.274117118Z version [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.279597609Z cluster-scoped-resources/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.27962122Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.27962696Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.279667861Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.279754852Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.279764093Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.279811894Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.279889375Z namespaces/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.279898245Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.279955306Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280036678Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280088289Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28013942Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280209882Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280311994Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280409226Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280504657Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280568799Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28062044Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280681551Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280730842Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280822804Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280878865Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.280928286Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.281022368Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.281080879Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28112818Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.281284703Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.281375895Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.281467977Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.281568509Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.281660361Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.281808144Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282017228Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282080229Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28212668Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282200192Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282244673Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282302164Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282360825Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282454307Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282513478Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282563619Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28261264Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282666051Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282717562Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282767703Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282818964Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282862525Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282923887Z namespaces/kuadrant-system/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.282979548Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.283044549Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28309334Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.283141061Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.283210883Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.283312034Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.283670662Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.283998908Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284054009Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28409303Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284150742Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284226843Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284325565Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284382306Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284433867Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284529259Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28458385Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284638131Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284813525Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.284964608Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.285505379Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.285602831Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.285996889Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.286140402Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.286407897Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28655626Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.286591511Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.286739094Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.286786995Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.286840566Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.286895337Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.286997889Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.28704698Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287091371Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287147222Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287219674Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287278815Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287287375Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287340106Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino-7cf4765b85-w4xqt.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287425978Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287434638Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287439918Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.287501899Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.301938792Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302021864Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302070085Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302128886Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/authorino-operator-56fb6f6856-t4rz2.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302256238Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302272779Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302278109Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.30230929Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302499583Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302575085Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302612606Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302674007Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/dns-operator-controller-manager-ccbdf686-447cz.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302773449Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302781849Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302785789Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.30282827Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.302951843Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303036414Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303081625Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303136136Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin-598f86468b-ncggk.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303245849Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303257989Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303262219Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.30330528Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303391901Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303472373Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303529084Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303582996Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/kuadrant-operator-controller-manager-66866f5fcc-5z76z.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303722328Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303730159Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/manager/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.303734039Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/manager/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.30378026Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/manager/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.318279134Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/manager/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.322711173Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.322781695Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador-limitador-85d9db4d5d-fcvd7.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.322860316Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.322869836Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.322875007Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.322917128Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323006729Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323083591Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323100351Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323181353Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/limitador-operator-controller-manager-774d94476f-4cd9s.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323274845Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323285185Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323290095Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323321476Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323473209Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323556661Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323594341Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323642522Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323687003Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323735404Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323801775Z namespaces/openshift-ingress/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323863847Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323922138Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.323967679Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.32401328Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324054241Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324145733Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324433758Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324653993Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324693624Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324737185Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324805566Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324834387Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324932638Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.324978319Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.32501887Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.325099982Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.325144303Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.325215434Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.325646863Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.325765995Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.32598604Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.326104042Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.32648813Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.326637393Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.326868008Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.326948829Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.32699814Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327087152Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327125463Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327193584Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327230035Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327323827Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327366998Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327410749Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.32746471Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327527911Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327569092Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327577942Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327615433Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/istiod-openshift-gateway-5677ccb49b-8bfqq.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327720705Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327728835Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327733135Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.327762596Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.341494064Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.341556836Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.341578396Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.341634627Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/openshift-ai-inference-openshift-default-8699b59bbc-spfsb.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.342145647Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.342158148Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.342163328Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.34224616Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343351412Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343434204Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343494085Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router-default-b46967c9f-ptz2d.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343602287Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343612127Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343616937Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343639928Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/current.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.34375375Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/previous.insecure.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343823612Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/previous.log [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343862642Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343902643Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343939954Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.343993055Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.352245642Z [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.352263423Z sent 2,829 bytes received 1,064,003 bytes 711,221.33 bytes/sec [must-gather] [must-gather-k76f6] OUT 2026-07-27T19:14:36.352268623Z total size is 17,951,655 speedup is 16.83 [must-gather] [must-gather ] OUT 2026-07-27T19:14:36.401107293Z namespace/openshift-must-gather-sth4s deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 328368b6-397a-4aea-a1e7-1353e5a2e7f8 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-27T19:14:36.522872342Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 328368b6-397a-4aea-a1e7-1353e5a2e7f8 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-27T19:14:36.547097974Z namespace/openshift-must-gather-r8n2x created [must-gather] [must-gather ] OUT 2026-07-27T19:14:36.555040385Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-42457 created [must-gather] [must-gather ] OUT 2026-07-27T19:14:36.580188585Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:37.802852815Z [disk usage checker] Started [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:37.806361165Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:38.014966157Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:38.162335674Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:38.253376367Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:38.345152249Z Gathering data for ns/default... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:38.688277648Z Gathering data for ns/openshift... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:39.113552464Z Gathering data for ns/kube-system... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:39.991152026Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.330681888Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.330721293Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.330721293Z namespaces "assisted-installer" not found [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.705640647Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.713858984Z Executing Istio gather script [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.714356879Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.714687310Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.717065392Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.719954716Z INFO: Collecting host service logs for crio [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.720398711Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.720778734Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.721123057Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.721463757Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.721787171Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.722137500Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.722486432Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.722827224Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.723169546Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.723663286Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.724094324Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.751366259Z INFO: Gathering HAProxy config files [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:40.751849502Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.296255950Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.524907489Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.864867559Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.938564752Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-jvqln [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.939495261Z insights-runtime-extractor-9jp2v [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.939531985Z insights-runtime-extractor-cz9hj [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.939554316Z insights-runtime-extractor-dd9n4 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.953692165Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:41.993316251Z No resources found [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.162642437Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.162727632Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.165373278Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.579996434Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.701941116Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.701941116Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.740035567Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.746806849Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.926223189Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:42.938871376Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.047389639Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.072193894Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.084963648Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.121691205Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.125844270Z INFO: 'previous-logs' folder not found on ip-10-0-131-114.ec2.internal, skipping... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.127067655Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.279204244Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.529107435Z No resources found [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.560829926Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.639505262Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.672097945Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.791298502Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.828533454Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.855525593Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.943627097Z tar: Removing leading `/' from member names [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:43.948511554Z INFO: HAProxy config collection complete. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.072029982Z INFO: 'previous-logs' folder not found on ip-10-0-131-74.ec2.internal, skipping... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.135650038Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.140879673Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.362285580Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.365902931Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.377108873Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.432755627Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.601606900Z INFO: 'previous-logs' folder not found on ip-10-0-135-40.ec2.internal, skipping... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.601713631Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.601762671Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.704249665Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.790554349Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.793751011Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:44.897750464Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.005912750Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.269659593Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.298067639Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.304109907Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.365943713Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.447707512Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.579311423Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.732208784Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.736481186Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:45.785450650Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.003260992Z INFO: INTERCONNECT MODE [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.003337552Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.025127978Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.040609259Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.139092280Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.160532948Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.213921414Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.224661934Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.300509882Z INFO: Gathering OVN_Northbound from ovnkube-node-74rhp... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.300863500Z INFO: Gathering OVN_Northbound from ovnkube-node-gwwvs... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.301156774Z INFO: Gathering OVN_Northbound from ovnkube-node-nld5j... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.301512453Z INFO: Gathering OVN_Southbound from ovnkube-node-74rhp... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.301807031Z INFO: Gathering OVN_Southbound from ovnkube-node-gwwvs... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.302283448Z INFO: Gathering OVN_Southbound from ovnkube-node-nld5j... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.380331220Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.511522285Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.631990923Z tar: Removing leading `/' from member names [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.636724035Z tar: Removing leading `/' from member names [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.701810063Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.737216279Z tar: Removing leading `/' from member names [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.776782458Z tar: Removing leading `/' from member names [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.785217959Z tar: Removing leading `/' from member names [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.814547229Z tar: Removing leading `/' from member names [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.908519943Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:46.969907409Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.196331095Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.308609708Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.409364374Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.409576114Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.409894548Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.412022242Z 7.0M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.414555999Z ovnk_database_store/ [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.414555999Z ovnk_database_store/ovnkube-node-gwwvs_sbdb [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.456578549Z ovnk_database_store/ovnkube-node-74rhp_sbdb [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.501262142Z ovnk_database_store/ovnkube-node-74rhp_nbdb [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.519964794Z ovnk_database_store/ovnkube-node-nld5j_nbdb [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.534802514Z ovnk_database_store/ovnkube-node-gwwvs_nbdb [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.552620784Z ovnk_database_store/ovnkube-node-nld5j_sbdb [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.605681039Z INFO: Network log collection complete. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.646359715Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.709077750Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:47.980652374Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:48.102070219Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:48.278736406Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:48.281978856Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:48.587256636Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:48.733521299Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:48.900932709Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.044109492Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.065184760Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.173658697Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.213834540Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.371041760Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.528795568Z Gathering data for ns/openshift-console... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.622858432Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.626912319Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:49.882283618Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.100683876Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.120837466Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.123705437Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.338399542Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.362528692Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.575646741Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.578599043Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.664416400Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:50.786693318Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.002573964Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.009569894Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.011880926Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.195220094Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.425548854Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.429372016Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.524146075Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.642400714Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.642718253Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.872930253Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:51.876267139Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.069699145Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.198153789Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.309555755Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.312269482Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.577354497Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.595784613Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.690666398Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.844708411Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:52.848211631Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.064310121Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.107124816Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.287743146Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.290364415Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.467752125Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.599670656Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.674562169Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.677454058Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.838540513Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:53.893686798Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.081808531Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.128761402Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.131331109Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.291850744Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.497847594Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.501756678Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.695094902Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.908125881Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.911720208Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:54.969888724Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:55.087511749Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:55.272200747Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:55.274855727Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:55.448666275Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:55.653935245Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:55.657816942Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:55.849370331Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.048340507Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.050899497Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.128944971Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.240493229Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.447086361Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.449604044Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.660803520Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.864505269Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:56.867616250Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.044969973Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.119864284Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.264234659Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.267836847Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.275390753Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.474578846Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.584981355Z Collecting performance related data for node ip-10-0-131-114.ec2.internal [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.597195286Z Collecting performance related data for node ip-10-0-131-74.ec2.internal [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.611956895Z Collecting performance related data for node ip-10-0-135-40.ec2.internal [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.765768356Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.769710498Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.819113867Z Gathering data for ns/kserve... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:57.958595058Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:58.113783319Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:58.202162827Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:58.350741385Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:58.603637544Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:59.422859813Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:59.613789136Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:59.760337332Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:14:59.887270747Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:00.116859573Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:00.839653886Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:01.018523330Z Gathering data for ns/openshift-config... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:01.044372398Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:01.197692869Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:01.367819068Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:01.726953239Z Done executing Istio gather script [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:01.844380466Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:02.159367463Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:02.471548200Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:02.831518544Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:03.118856860Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:03.176240465Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:03.481620233Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:03.843730555Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:04.258496673Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:04.620309980Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:05.147282350Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:06.347547049Z Collecting kubelet logs for node ip-10-0-131-114.ec2.internal [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:06.349367203Z Collecting kubelet logs for node ip-10-0-131-74.ec2.internal [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:06.351183632Z Collecting kubelet logs for node ip-10-0-135-40.ec2.internal [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:06.813069366Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-r8n2x namespace [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:06.818202793Z INFO: Node performance data collection complete. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:06.835308729Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:08.124401224Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:08.228067927Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:08.563830901Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:09.105508647Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:09.505645294Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:09.874453952Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:10.174855470Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:10.684651906Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:11.049193905Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:11.473776020Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:11.784312695Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:12.103835225Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:12.867082293Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:13.128885898Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:13.196633318Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:13.568665457Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:14.011342660Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:15.128897746Z Wrote inspect data to must-gather. [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:15.128946451Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:15.128946451Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:15.267929876Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-nk8b4] POD 2026-07-27T19:15:15.459219744Z Caches written to disk [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.469557538Z waiting for gather to complete [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.473681583Z downloading gather output [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.769260229Z receiving incremental file list [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.791326248Z ./ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.791438571Z aggregated-discovery-api.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.791760187Z aggregated-discovery-apis.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.792811638Z event-filter.html [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.795719228Z timestamp [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.795797269Z version [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.801986985Z cluster-scoped-resources/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802001166Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802020396Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.80222044Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802305712Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802438105Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802508136Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802578897Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802696Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802799242Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802910944Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.802955255Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.803091348Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.80318493Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.803600978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.803626389Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.803860063Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.804122269Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.804351793Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.804490296Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.808468737Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.810320535Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.810467248Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.81055902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.810671622Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.810802075Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.811277615Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.811425068Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.811591081Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.811701003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.812582291Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.812799856Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.812918768Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.813104522Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.814266515Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.81449966Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.814660654Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.814820957Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.815096902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.815253015Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.81545307Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.815720315Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.816421649Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.816542642Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.816655354Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.816768786Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.816865648Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.816970881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.817129474Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.817341568Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.818118244Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.818266537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.8183944Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.82036243Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.820522003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.820684816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.820919721Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.821197387Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.821644436Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.821758638Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.821876421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.821980152Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.822088325Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.822228438Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.82233455Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.822478443Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.822610705Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.822721718Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.822985813Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.823113816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.823241238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.82379898Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.823919052Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.824035894Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.824281419Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.825030455Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.825134367Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.825350231Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.825459394Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.825622407Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.825736739Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.825890952Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.826023265Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.826141277Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.826300121Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.826438603Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.826607327Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.826733109Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.826885212Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.827102107Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.82724468Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.827395533Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.8277675Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.827935514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.830045227Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.830154809Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.830946525Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.831080528Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.831210671Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.831374514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.831508807Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.831709971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.831949966Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.832075628Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.832245732Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.832445646Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.832602219Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.832758842Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.832939486Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.833098089Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.837156412Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.837651502Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.838232034Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.838409567Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.838572281Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.838713683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.838870016Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.838989769Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.839087911Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.839909988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.840351977Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.844360318Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.844493781Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.844639054Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.844767387Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.844878109Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.844996291Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.845112954Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.84787897Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.850993503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.85619634Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.861518308Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.861671231Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.861961537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.86211909Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.862279653Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.862473017Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.86262288Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.862909346Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.863020348Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.863125221Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.863336935Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.863471008Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.8636001Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.863712293Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.863819345Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.863952847Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.864127941Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.864276564Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.864387526Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.864510789Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.864669532Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.864774074Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.864910467Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.865020039Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.865139642Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.865495929Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.865634852Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.865885657Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.866000299Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.866121811Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.866245294Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.868242665Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.868368727Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.86848123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.868608572Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.868717695Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.868880418Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.869013121Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.869195804Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.869330907Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.871012771Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.871156564Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.871313397Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.87146144Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.871588583Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.871766026Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.872020362Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.872552002Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.872828288Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.872944521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.873060173Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.873159605Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.873821898Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.874024112Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.874119525Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.875602765Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.875761868Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.875928581Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.876034243Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.876225707Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.876555494Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.876739588Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.876950782Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.8773707Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.877480253Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.877597135Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.877759819Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.877916862Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.878051794Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.878198797Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.878380511Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.878696678Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.878721218Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.87882729Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.878911702Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.878991834Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879072865Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879252329Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879347661Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879433243Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879533945Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879618957Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879707908Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879828841Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.879929823Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880029635Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880115416Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880220109Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880318331Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880403433Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880489884Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880580906Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880670078Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88076055Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880843571Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.880938993Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881029145Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881112067Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881217619Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881305731Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881392502Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881498685Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881607167Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.8817397Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881862122Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.881982305Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.882107527Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.882230879Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.882360672Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.882477024Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.882610597Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88273045Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.882863252Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.882981875Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.883108428Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88323509Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.883361793Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.883498605Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.883615908Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.883775951Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.883897464Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.884024786Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.884152899Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.884522146Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.884614438Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88471641Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.884808932Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.884921784Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885007696Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885128679Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885260611Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885360923Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885453565Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885548837Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885630019Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885725021Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885846753Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.885930745Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886023497Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886117319Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886234841Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886337763Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886429185Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886520297Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886611969Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886699611Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886798493Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886880684Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.886973626Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.887062448Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88715467Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.887279062Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.887372654Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.887462916Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.887576288Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88766529Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.887758322Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.887848924Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.887941586Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.888029828Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88812439Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.888244322Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.888340884Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.888452636Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.888542298Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88864248Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.888730182Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.888823504Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.888922436Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889013358Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.88910937Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889223682Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889326044Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889413466Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889506358Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889595509Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889683691Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889779063Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889898446Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.889945697Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.8901094Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.890274663Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.890376216Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.890464327Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.890798834Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.890905576Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89110978Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.891218693Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.891345085Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.891417797Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.891513889Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89160377Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.891690442Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.891786834Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.891875736Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.891967658Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89205517Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892141841Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892272664Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892354986Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892441198Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892523879Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892642452Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892690183Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892793265Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892883047Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.892973508Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89305892Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.893154892Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.893292255Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.893452458Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89353003Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.893632362Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.893722634Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.893821256Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.893947018Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89401734Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894134642Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894246094Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894333926Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894418898Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89451604Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894607932Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894693163Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894824296Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894870237Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894949318Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.894999319Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895051121Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895089081Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895152303Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895215314Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895279605Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895320126Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895384187Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895428268Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89549601Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89550575Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895566251Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895661833Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895752655Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895806106Z cluster-scoped-resources/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895812986Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.895854617Z cluster-scoped-resources/core/nodes/ip-10-0-131-114.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89602242Z cluster-scoped-resources/core/nodes/ip-10-0-131-74.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896160473Z cluster-scoped-resources/core/nodes/ip-10-0-135-40.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896289836Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896327527Z cluster-scoped-resources/core/persistentvolumes/pvc-47e956c4-9962-4d79-8479-81e5daeb1dba.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896396748Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896428259Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89648608Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896586112Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896671624Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896754995Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896839437Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.896923369Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.897078122Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.897184684Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.897298866Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.897391958Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89748441Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.897575302Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.897675404Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.897766246Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.897859798Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89794618Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898043672Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898129333Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898255746Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898344868Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89844035Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898535722Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898583153Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898671514Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898749826Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.898836438Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89892421Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899001671Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899145004Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899299377Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899398999Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89945695Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.89946321Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899517172Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899579063Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899589993Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899619904Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899719976Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899750976Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899825278Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899834628Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899889679Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.899979431Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900126714Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900223676Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900236486Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900296947Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900382599Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.9003936Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90043766Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900533472Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900555833Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900683645Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900704126Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900797548Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.900828668Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90089658Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90092453Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901063203Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901091164Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901150905Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901214886Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901260167Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901296608Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901371159Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90142085Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901492432Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901521042Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901610244Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901631055Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901696686Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901726836Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901804258Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90187459Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901951431Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.901999132Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.902099274Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.902107004Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.902156445Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.902261778Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.902303898Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.902600334Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.902706847Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90284781Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.902951321Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903064814Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903181936Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903273078Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903300499Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90335402Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903443702Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903542924Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903694777Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903820749Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903910401Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.903997413Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904084665Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904184237Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904278719Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90436012Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904446362Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904532764Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904617795Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904703837Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904796459Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904886061Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.904973113Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905061184Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905146956Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905257559Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905393031Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905429552Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905532134Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905613476Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905702648Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90584526Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.905943342Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906033014Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906136246Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906247449Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906335401Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906421592Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906510014Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906597336Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906684777Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906774939Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906864961Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.906958233Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907044905Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907138897Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907258589Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907348191Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907440393Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907543335Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907631307Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907724819Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907818881Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907905712Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.907997254Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908089086Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908184248Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908343881Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908351911Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908400563Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908478364Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908485174Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908529365Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908570086Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908576726Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908617997Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908667348Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908711839Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-114.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908809251Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-74.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908893313Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-135-40.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908929333Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.908980364Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909068036Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909115797Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909146838Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-f3c1039b9aa07e2d1e308d2ee9875c79920b53784a0c9903fa040900ccbf6b83.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90925137Z host_service_logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.90926375Z host_service_logs/masters/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909334831Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909394173Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909462044Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909532655Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909663628Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909772011Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909841802Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909922603Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.909982615Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910053886Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910119818Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910153118Z ingress_controllers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910161988Z ingress_controllers/default/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910182199Z ingress_controllers/default/router-default-b46967c9f-ptz2d/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910236Z ingress_controllers/default/router-default-b46967c9f-ptz2d/haproxy.config [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910353482Z insights-data/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910369583Z istio/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910432544Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.910602737Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911109758Z istio/event-filter.html [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911277591Z istio/timestamp [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911382903Z istio/cluster-scoped-resources/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911390713Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911394833Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911428944Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911519776Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911567167Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911636049Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91170592Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911750311Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.911948385Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.912867874Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.913061737Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.913254851Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91367577Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.913847783Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.914264862Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91514871Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.915367384Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.915509837Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91564345Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.915753612Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.915875255Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.916033168Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.916320094Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.916696052Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.916895105Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.917315064Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.917452027Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91758516Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.917756843Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.917766723Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.917817874Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.917865095Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.917875916Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.917944247Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918034609Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918182362Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918266973Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918320334Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918418607Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91859364Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918669482Z istio/namespaces/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918678002Z istio/namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918681882Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918690662Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918737533Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918830675Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918838415Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918883856Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918959487Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.918966218Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919010379Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91906785Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91907703Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919134201Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919247203Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919395516Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919526449Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.91957989Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919652162Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919709193Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919778154Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919825225Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.919931877Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920188423Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920484349Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.9205412Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920585701Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920660732Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920703583Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920787085Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920830396Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920892337Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.920974628Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.92105611Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.921097811Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.921666363Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.921768815Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922002909Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922076881Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922350377Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922485679Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922694024Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922797216Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922846617Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922922968Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.922933988Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.92298926Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923066991Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923115322Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923221334Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923245005Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923349437Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923433869Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923466819Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923548011Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923594852Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923618842Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923665314Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923759015Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.923887518Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.92397793Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.92401633Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.924069242Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.924076842Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.924117103Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/istiod-openshift-gateway-5677ccb49b-8bfqq.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.924253005Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.924264126Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.924268276Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.924295786Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.933539064Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.933593625Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.933616106Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.933677107Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/openshift-ai-inference-openshift-default-8699b59bbc-spfsb.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.93381528Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.933832821Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.933841351Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.933895092Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.934996884Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935114776Z istio/namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935136807Z istio/namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router-default-b46967c9f-ptz2d.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.93527978Z istio/namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.93529679Z istio/namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935304931Z istio/namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935343281Z istio/namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935453594Z istio/namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935536735Z istio/namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935585986Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935630647Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935672698Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935724109Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.93577565Z machine_config_ondisk/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.93578729Z machine_config_termination_logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935795441Z monitoring/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.93579955Z monitoring/alertmanager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935842321Z monitoring/alertmanager/status.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935940653Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.935988324Z monitoring/prometheus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.936026385Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.936123117Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.936209799Z monitoring/prometheus/rules.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.936985915Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.936999155Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.937048056Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.937980465Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938046766Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938093967Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938183669Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938281371Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938367573Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938414734Z monitoring/prometheus/status/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938434344Z monitoring/prometheus/status/config.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938884803Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.938946685Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939040076Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939288961Z namespaces/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939308722Z namespaces/cert-manager-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939313852Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939322492Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939347313Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939427954Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939440135Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939499806Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.93968792Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.939936715Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940080908Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940332233Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940533967Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940576398Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-7wqxq.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940739671Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940795642Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940865124Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940886134Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.940969826Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.941009267Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.941074028Z namespaces/cert-manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.941085488Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.941094468Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.941139589Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.941333233Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.941523867Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94166187Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.941867804Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942103809Z namespaces/default/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94214295Z namespaces/default/default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942237732Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942258262Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942325403Z namespaces/default/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942359534Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942444166Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942518997Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94262306Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94264781Z namespaces/default/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942700721Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942723202Z namespaces/default/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942787513Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.942985817Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943043148Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943064638Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943192251Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943269713Z namespaces/default/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943303813Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943409885Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943523408Z namespaces/default/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943694781Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943772223Z namespaces/default/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943848384Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.943972757Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94413494Z namespaces/default/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944245053Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944282303Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944366465Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944410786Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944479867Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944522158Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94460887Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944686811Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944734742Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944767103Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944821054Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944858455Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944906846Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944915886Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.944945457Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94510203Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.945342115Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.945487058Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.945693312Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.945884996Z namespaces/default/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.945916376Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.945979828Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.946012178Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94608082Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94608967Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94610277Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.946152841Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.946330675Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.946504339Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.946649062Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.946843305Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.947042979Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94705356Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.947062Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.947111661Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.947294555Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.947579981Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.947717253Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.947918697Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.948107981Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.948115751Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.948125952Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.948189413Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.948363346Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94855482Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.948696893Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.948885757Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.949097051Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.949105312Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.949114302Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.949163973Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.949343386Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.94952839Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.949667693Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.949865977Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.950077121Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.950086961Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.950099092Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.950148663Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.950324876Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.95050925Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.950651363Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.950845427Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951038091Z namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951051131Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951056541Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951068441Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951197024Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951221664Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951291016Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951377098Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951427289Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951446659Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.95149769Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951660693Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.951839947Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.95197616Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.952187064Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.952396319Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.952405179Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.952413969Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.95246086Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.952618833Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.952793876Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.952928099Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.953119983Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.953502311Z namespaces/kserve/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.953540782Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.953620033Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.953653004Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.953715475Z namespaces/kserve/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.953747786Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.953839598Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954036482Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954239356Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954282617Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954322428Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954382149Z namespaces/kserve/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.95442883Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954503091Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954592943Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954633494Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954710626Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954742706Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954748207Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954790157Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954887329Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.954970881Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.955057013Z namespaces/kserve/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.955103674Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.955369129Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.955480651Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.955821228Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.95589878Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.956159165Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.956305338Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.956546543Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.956651275Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.956695746Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.956785648Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.956816449Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.95688756Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.956935451Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957027313Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957104274Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957157775Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957237627Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957312639Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957355109Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957445911Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957492692Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957524533Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957597854Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957606934Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957667306Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957813829Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.957991853Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958128165Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.9583553Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958560124Z namespaces/kserve/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958568614Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-nxnm8/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958620755Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-nxnm8/kserve-controller-manager-ff96d5959-nxnm8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958723537Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-nxnm8/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958731708Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-nxnm8/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958745758Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-nxnm8/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958791869Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-nxnm8/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.958920811Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-nxnm8/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.959012413Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-nxnm8/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.959032874Z namespaces/kserve/pods/llmisvc-controller-manager-9c4f55bbb-9kklv/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.959086995Z namespaces/kserve/pods/llmisvc-controller-manager-9c4f55bbb-9kklv/llmisvc-controller-manager-9c4f55bbb-9kklv.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.959194487Z namespaces/kserve/pods/llmisvc-controller-manager-9c4f55bbb-9kklv/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.959214997Z namespaces/kserve/pods/llmisvc-controller-manager-9c4f55bbb-9kklv/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.959222578Z namespaces/kserve/pods/llmisvc-controller-manager-9c4f55bbb-9kklv/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.959258068Z namespaces/kserve/pods/llmisvc-controller-manager-9c4f55bbb-9kklv/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961103316Z namespaces/kserve/pods/llmisvc-controller-manager-9c4f55bbb-9kklv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961156567Z namespaces/kserve/pods/llmisvc-controller-manager-9c4f55bbb-9kklv/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.96129443Z namespaces/kserve/pods/model-serving-api-7b954ffc95-5r24t/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961335661Z namespaces/kserve/pods/model-serving-api-7b954ffc95-5r24t/model-serving-api-7b954ffc95-5r24t.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961425532Z namespaces/kserve/pods/model-serving-api-7b954ffc95-5r24t/server/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961439743Z namespaces/kserve/pods/model-serving-api-7b954ffc95-5r24t/server/server/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961448503Z namespaces/kserve/pods/model-serving-api-7b954ffc95-5r24t/server/server/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961492024Z namespaces/kserve/pods/model-serving-api-7b954ffc95-5r24t/server/server/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961585006Z namespaces/kserve/pods/model-serving-api-7b954ffc95-5r24t/server/server/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961649137Z namespaces/kserve/pods/model-serving-api-7b954ffc95-5r24t/server/server/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961684378Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-ddz92/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961726569Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-ddz92/odh-model-controller-75d9cdbddf-ddz92.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961868301Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-ddz92/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961877372Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-ddz92/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961888842Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-ddz92/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.961939893Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-ddz92/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99074982Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-ddz92/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.990824341Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-ddz92/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.990882992Z namespaces/kserve/pods/s3-init-zggbs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.990940064Z namespaces/kserve/pods/s3-init-zggbs/s3-init-zggbs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991035515Z namespaces/kserve/pods/s3-init-zggbs/download-hf-model/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991044195Z namespaces/kserve/pods/s3-init-zggbs/download-hf-model/download-hf-model/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991049676Z namespaces/kserve/pods/s3-init-zggbs/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991119737Z namespaces/kserve/pods/s3-init-zggbs/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991240039Z namespaces/kserve/pods/s3-init-zggbs/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991308871Z namespaces/kserve/pods/s3-init-zggbs/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991329001Z namespaces/kserve/pods/s3-init-zggbs/s3-init/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991333941Z namespaces/kserve/pods/s3-init-zggbs/s3-init/s3-init/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991351422Z namespaces/kserve/pods/s3-init-zggbs/s3-init/s3-init/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991399743Z namespaces/kserve/pods/s3-init-zggbs/s3-init/s3-init/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991695139Z namespaces/kserve/pods/s3-init-zggbs/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99174282Z namespaces/kserve/pods/s3-init-zggbs/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991786681Z namespaces/kserve/pods/seaweedfs-64568bcd49-q5r7l/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991825691Z namespaces/kserve/pods/seaweedfs-64568bcd49-q5r7l/seaweedfs-64568bcd49-q5r7l.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991908473Z namespaces/kserve/pods/seaweedfs-64568bcd49-q5r7l/seaweedfs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991924033Z namespaces/kserve/pods/seaweedfs-64568bcd49-q5r7l/seaweedfs/seaweedfs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991935374Z namespaces/kserve/pods/seaweedfs-64568bcd49-q5r7l/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.991951424Z namespaces/kserve/pods/seaweedfs-64568bcd49-q5r7l/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992188029Z namespaces/kserve/pods/seaweedfs-64568bcd49-q5r7l/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992277251Z namespaces/kserve/pods/seaweedfs-64568bcd49-q5r7l/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992320492Z namespaces/kserve/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992336772Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992396013Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992431874Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992536146Z namespaces/kuadrant-system/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992593577Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992668949Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992697509Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992773411Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992805071Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.992896763Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993187919Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993475145Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993520906Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993560027Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993626208Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993659629Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99374881Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993780831Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993837692Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993915844Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993953565Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.993990726Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.994024986Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.994121948Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99423149Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.994315032Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.994363623Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.994407934Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.994558767Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99467892Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.995123949Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.995235631Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.995579368Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.995703721Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.995911075Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996027927Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996057498Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99619295Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996216601Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996279032Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996323133Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996405165Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996481656Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996529447Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996562108Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996616239Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996636849Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996709271Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996720321Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996775362Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.996986616Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.997222021Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.997370644Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.997576999Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.997796303Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.997834734Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-wb5zl.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998198451Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998218502Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998320114Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998405985Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998481627Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998533598Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998583259Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99863145Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998682381Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998775733Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.998930896Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.999027928Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99910993Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.99911821Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.999181341Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino-7cf4765b85-w4xqt.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.999270123Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.999279983Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.999288223Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:16.999338764Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012263568Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012334969Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-w4xqt/authorino/authorino/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0123526Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012414241Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/authorino-operator-56fb6f6856-t4rz2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012508843Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012519113Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012531793Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012580964Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012765738Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012834039Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-t4rz2/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.01289065Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.012928931Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/dns-operator-controller-manager-ccbdf686-447cz.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013023273Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013030403Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013041314Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013098125Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013252078Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013324329Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-447cz/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.01337372Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013424091Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin-598f86468b-ncggk.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013501213Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013514193Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013523413Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013572514Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013644786Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013715667Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ncggk/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013742308Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013788829Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/kuadrant-operator-controller-manager-66866f5fcc-5z76z.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013907621Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013915961Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013920331Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.013957592Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.024434375Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-5z76z/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.027876006Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.027930297Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador-limitador-85d9db4d5d-fcvd7.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.02810817Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028122421Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028130471Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028145541Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028250513Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028331325Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-fcvd7/limitador/limitador/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028350935Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028413667Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/limitador-operator-controller-manager-774d94476f-4cd9s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028518889Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028528669Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028534009Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.02856069Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028747613Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028794094Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-4cd9s/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028850416Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028886266Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028944038Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.028975128Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.02904292Z namespaces/kube-node-lease/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029050629Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.02905495Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029102021Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-114.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029215873Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-74.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029314265Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-135-40.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029356466Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029363556Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029427197Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029624361Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029839766Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.029980868Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.030219043Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.030446148Z namespaces/kube-public/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.030454128Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.030460478Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.030511929Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.030674293Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.030854226Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031003209Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031219544Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031461549Z namespaces/kube-system/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.03151216Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031592011Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031634252Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031699334Z namespaces/kube-system/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031741414Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031874127Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.031952368Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.03203213Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032089451Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032138552Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032212414Z namespaces/kube-system/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032267305Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032356167Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032418438Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032447089Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.03253506Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032577431Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032584552Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032649263Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-7v5ojw6os24v4f4ticcadckxpu.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032725944Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032821316Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.032922218Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.033069551Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.033133503Z namespaces/kube-system/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.033193984Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.033380768Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.033464579Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.033733225Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.033809186Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034081202Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034229175Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034572892Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034628603Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034679014Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034734045Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034749836Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034885168Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.034939449Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035027141Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035101303Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035146564Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035222715Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035266846Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035326147Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035390469Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035407029Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035422449Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035603403Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.035820518Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.03594745Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.036139964Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.036378529Z namespaces/kube-system/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.036391749Z namespaces/kube-system/pods/global-pull-secret-syncer-8kxfj/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.03643521Z namespaces/kube-system/pods/global-pull-secret-syncer-8kxfj/global-pull-secret-syncer-8kxfj.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.036511001Z namespaces/kube-system/pods/global-pull-secret-syncer-8kxfj/global-pull-secret-syncer/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.036518852Z namespaces/kube-system/pods/global-pull-secret-syncer-8kxfj/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.036523022Z namespaces/kube-system/pods/global-pull-secret-syncer-8kxfj/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.036574513Z namespaces/kube-system/pods/global-pull-secret-syncer-8kxfj/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.03694326Z namespaces/kube-system/pods/global-pull-secret-syncer-8kxfj/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037012752Z namespaces/kube-system/pods/global-pull-secret-syncer-8kxfj/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037078973Z namespaces/kube-system/pods/global-pull-secret-syncer-qxkrc/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037112464Z namespaces/kube-system/pods/global-pull-secret-syncer-qxkrc/global-pull-secret-syncer-qxkrc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037187455Z namespaces/kube-system/pods/global-pull-secret-syncer-qxkrc/global-pull-secret-syncer/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037201926Z namespaces/kube-system/pods/global-pull-secret-syncer-qxkrc/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037206146Z namespaces/kube-system/pods/global-pull-secret-syncer-qxkrc/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037254047Z namespaces/kube-system/pods/global-pull-secret-syncer-qxkrc/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037614764Z namespaces/kube-system/pods/global-pull-secret-syncer-qxkrc/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037672015Z namespaces/kube-system/pods/global-pull-secret-syncer-qxkrc/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037723376Z namespaces/kube-system/pods/global-pull-secret-syncer-zzcb2/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037738417Z namespaces/kube-system/pods/global-pull-secret-syncer-zzcb2/global-pull-secret-syncer-zzcb2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037816848Z namespaces/kube-system/pods/global-pull-secret-syncer-zzcb2/global-pull-secret-syncer/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037826078Z namespaces/kube-system/pods/global-pull-secret-syncer-zzcb2/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037832498Z namespaces/kube-system/pods/global-pull-secret-syncer-zzcb2/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.037848119Z namespaces/kube-system/pods/global-pull-secret-syncer-zzcb2/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.038253077Z namespaces/kube-system/pods/global-pull-secret-syncer-zzcb2/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.038308598Z namespaces/kube-system/pods/global-pull-secret-syncer-zzcb2/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.038335618Z namespaces/kube-system/pods/konnectivity-agent-6lb2p/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.03838153Z namespaces/kube-system/pods/konnectivity-agent-6lb2p/konnectivity-agent-6lb2p.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.038471181Z namespaces/kube-system/pods/konnectivity-agent-6lb2p/konnectivity-agent/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.038486852Z namespaces/kube-system/pods/konnectivity-agent-6lb2p/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.038499242Z namespaces/kube-system/pods/konnectivity-agent-6lb2p/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.038513782Z namespaces/kube-system/pods/konnectivity-agent-6lb2p/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.039847009Z namespaces/kube-system/pods/konnectivity-agent-6lb2p/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.039906131Z namespaces/kube-system/pods/konnectivity-agent-6lb2p/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.039952182Z namespaces/kube-system/pods/konnectivity-agent-cq6fr/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.039995563Z namespaces/kube-system/pods/konnectivity-agent-cq6fr/konnectivity-agent-cq6fr.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.040099465Z namespaces/kube-system/pods/konnectivity-agent-cq6fr/konnectivity-agent/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.040109655Z namespaces/kube-system/pods/konnectivity-agent-cq6fr/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.040117115Z namespaces/kube-system/pods/konnectivity-agent-cq6fr/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.040156976Z namespaces/kube-system/pods/konnectivity-agent-cq6fr/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.041216927Z namespaces/kube-system/pods/konnectivity-agent-cq6fr/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.041278649Z namespaces/kube-system/pods/konnectivity-agent-cq6fr/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.04135521Z namespaces/kube-system/pods/konnectivity-agent-xqqgl/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.041395971Z namespaces/kube-system/pods/konnectivity-agent-xqqgl/konnectivity-agent-xqqgl.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.041470602Z namespaces/kube-system/pods/konnectivity-agent-xqqgl/konnectivity-agent/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.041477913Z namespaces/kube-system/pods/konnectivity-agent-xqqgl/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.041482113Z namespaces/kube-system/pods/konnectivity-agent-xqqgl/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.041498823Z namespaces/kube-system/pods/konnectivity-agent-xqqgl/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.042560365Z namespaces/kube-system/pods/konnectivity-agent-xqqgl/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.042638856Z namespaces/kube-system/pods/konnectivity-agent-xqqgl/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.042658017Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.042708468Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.042792469Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal/haproxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.042801009Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.04280828Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.042852421Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043184377Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043258029Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.04330959Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043354841Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043431942Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal/haproxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043439102Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043443142Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043501024Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043734868Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.04380468Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-74.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043853961Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043891732Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043968973Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal/haproxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043978424Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.043982504Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044033085Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044330021Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044389842Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044433343Z namespaces/kube-system/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044470374Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044520435Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044534605Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044541985Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044650817Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044713028Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044757839Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.04480959Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044851711Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044898522Z namespaces/observability/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044909183Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044917673Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.044966384Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.045113427Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.045321681Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.045460144Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.045666288Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.045906623Z namespaces/open-cluster-management-285f140b-bfbb-4f11-b960-0b299617d/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.045917323Z namespaces/open-cluster-management-285f140b-bfbb-4f11-b960-0b299617d/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.045925003Z namespaces/open-cluster-management-285f140b-bfbb-4f11-b960-0b299617d/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.045969024Z namespaces/open-cluster-management-285f140b-bfbb-4f11-b960-0b299617d/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.046162648Z namespaces/open-cluster-management-285f140b-bfbb-4f11-b960-0b299617d/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.046377933Z namespaces/open-cluster-management-285f140b-bfbb-4f11-b960-0b299617d/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.046521475Z namespaces/open-cluster-management-285f140b-bfbb-4f11-b960-0b299617d/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.04672636Z namespaces/open-cluster-management-285f140b-bfbb-4f11-b960-0b299617d/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.046942394Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.046953894Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.046958984Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.046987235Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.047079847Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.047118828Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.047139358Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.047188919Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.047369353Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.047573457Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0477192Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.047920484Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048161239Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.04822296Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048293152Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048325532Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048399644Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048443844Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048534246Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048605108Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048687329Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0487177Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048770371Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048811572Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048856203Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048933194Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.048980665Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.049010686Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.049096688Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.049151509Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.04921716Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.050356193Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.050442285Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.050518717Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.050592808Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05067587Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.050796792Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.050979886Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051052868Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051099669Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05115425Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051216881Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051274882Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051323143Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051403325Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051478506Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051554988Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051597859Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051668Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051705731Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051776652Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051807613Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.051876234Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.052066678Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.052108459Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.052275082Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.052454836Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.052588879Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.052787953Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.052972477Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053015068Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053052138Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053106079Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053241242Z namespaces/openshift-apiserver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053289033Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053362345Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053401365Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053464267Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053510108Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053592469Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053666861Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053747643Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053792264Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053821054Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053885415Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.053921636Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054003658Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054056669Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05410617Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054185661Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054284143Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054324104Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054423166Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054507088Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05457996Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054662281Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054741793Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.054863895Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055017808Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055072999Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05511935Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055179281Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055245023Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055303964Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055330995Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055423166Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055509558Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05556628Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05559516Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055664982Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055695202Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055752893Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055762113Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055822445Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.055969708Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056157541Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056316465Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056514239Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056704253Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056737863Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056796505Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056830585Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056885096Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056894047Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056898367Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.056952928Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.057023249Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05703651Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.057091331Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.057263234Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.057449878Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.057584821Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.057794785Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.057998079Z namespaces/openshift-authentication/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058006549Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058015479Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058065611Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058278965Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058470449Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058608102Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058807186Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058991519Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.058999899Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.05900511Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.059051451Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.059118502Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.059138222Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.059187373Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.059348586Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.059550451Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.059693674Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.059883977Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060289486Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060300956Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060305396Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060347577Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060411938Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060419038Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060469429Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060536911Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060543831Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060592962Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060745225Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.060921829Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061066481Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061283236Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06149261Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061532801Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061603632Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061624633Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061708085Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061758535Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061827027Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.061906049Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06199283Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062027941Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062063422Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062108103Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062141783Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062254006Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062279396Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062329967Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062410399Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0624589Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06246986Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062511511Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062598383Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062639513Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062767356Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.062850578Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06296795Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063066502Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063160604Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063323118Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063610333Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063668925Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063678985Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063750096Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063787897Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063840278Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063866529Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06395774Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.063998031Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064081173Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064164135Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064229506Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064285727Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064304238Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064363049Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06440953Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06441809Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064455021Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064607304Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.064784717Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06492181Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065116364Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065334048Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065353659Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06542706Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06543791Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065445531Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065512512Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065556853Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065607514Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065652575Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065737417Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065782338Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065849879Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06588764Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065967311Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.065996792Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066125645Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066221666Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066314888Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066354839Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0663953Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066451561Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066500352Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066576744Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066637865Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066668956Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066765548Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066805338Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066812689Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06686825Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.066951372Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.067038103Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.067150116Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.067312189Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0673603Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.067459342Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.067547593Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.067897231Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.067971822Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068189526Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068338069Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068453322Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068491353Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068539944Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068596225Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068637446Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068676476Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068725237Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068766788Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068813609Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068887371Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.068964212Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069002083Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069053004Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069088415Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069136136Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069242098Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069256538Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.06931446Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069475433Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069668267Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.069801089Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070020834Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070274449Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070288809Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07033021Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/aws-ebs-csi-driver-node-m82gn.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070452263Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-driver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070599916Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-driver/csi-driver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070609366Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-driver/csi-driver/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070651197Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070747969Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07082041Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070861661Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-liveness-probe/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070870571Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070876011Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.070916582Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071007364Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071085256Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071122676Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-node-driver-registrar/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071129246Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071141717Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071232068Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07131478Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071387442Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m82gn/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071419662Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071478433Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/aws-ebs-csi-driver-node-q4xrx.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071597376Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-driver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071605426Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-driver/csi-driver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071615756Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-driver/csi-driver/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071671128Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07177604Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071832811Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071851471Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-liveness-probe/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071857711Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071864441Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.071925613Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072020335Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072087476Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072106846Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-node-driver-registrar/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072115577Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072124757Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072177558Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07228617Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072362182Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q4xrx/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072415193Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072450784Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/aws-ebs-csi-driver-node-rtrlp.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072549195Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-driver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072558846Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-driver/csi-driver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072562856Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-driver/csi-driver/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072895153Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.072986474Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073055286Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073102407Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-liveness-probe/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073111587Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073117957Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073156608Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07328066Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073351282Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073398713Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-node-driver-registrar/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073405613Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073409793Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073460884Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073554646Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073618517Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-rtrlp/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073671368Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073702649Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07375475Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07376398Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073813101Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073879513Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073917883Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.073972574Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074015725Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074081206Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074191359Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074208699Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074339252Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074399253Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074409303Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074447774Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074510645Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074530296Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074566546Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07472606Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.074913574Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075047446Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075274361Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075482425Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075549107Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075635078Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075683169Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075817442Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075853633Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.075988775Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076066497Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076139419Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076182239Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076254151Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076324002Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076374003Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076416964Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076466135Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076548457Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076592658Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076627138Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07671283Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076920044Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076932455Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.076987976Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.077080208Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.077125008Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.078041567Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.078154629Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.078327843Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.078411325Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.078614749Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.078761922Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.078914915Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079035027Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079074108Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079119379Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07915758Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079254492Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079284592Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079353474Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079409065Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079508607Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079593019Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.07965031Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079708621Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079772282Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079830784Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079903135Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.079952446Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.080000197Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.080047678Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.080095879Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08011628Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.080187341Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.080401455Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.080596489Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.080744272Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.080951137Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081292023Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081302564Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7hkzw/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081351034Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7hkzw/tuned-7hkzw.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081459917Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7hkzw/tuned/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081467447Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7hkzw/tuned/tuned/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081471497Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7hkzw/tuned/tuned/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081524648Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7hkzw/tuned/tuned/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08163676Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7hkzw/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081709922Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7hkzw/tuned/tuned/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081784883Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-k5v8t/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081821644Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-k5v8t/tuned-k5v8t.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081912266Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-k5v8t/tuned/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081919186Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-k5v8t/tuned/tuned/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081923056Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-k5v8t/tuned/tuned/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.081972287Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-k5v8t/tuned/tuned/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082066509Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-k5v8t/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082151421Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-k5v8t/tuned/tuned/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082161761Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zgsmw/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082256143Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zgsmw/tuned-zgsmw.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082335715Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zgsmw/tuned/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082342955Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zgsmw/tuned/tuned/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082347125Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zgsmw/tuned/tuned/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082384486Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zgsmw/tuned/tuned/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082500728Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zgsmw/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08257127Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-zgsmw/tuned/tuned/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08260337Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082683812Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082736433Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082784994Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082820444Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082829415Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082898716Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-114.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.082983518Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-74.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08306823Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-135-40.ec2.internal.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.083118941Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.083159671Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.083379866Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.083461098Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08355259Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08356821Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.083648901Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.083685642Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.083770304Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.083916647Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084035189Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084106701Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084148512Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084224743Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084282184Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084357976Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084419457Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084457048Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08453571Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084610901Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084649102Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084858926Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.084941658Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085080481Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085162122Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085329176Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085469409Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085633442Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085681163Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085701593Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085781945Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085811876Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085892367Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.085940188Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08601846Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086096131Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086140272Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086191983Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086263945Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086299555Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086377317Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086412848Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086464959Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086503639Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0865559Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086566301Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086605861Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086749884Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.086934358Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.087082551Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.087295326Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.087801016Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.087818576Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.087833127Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator-6748489656-tr78b.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.087968109Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator-watch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0879768Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08800764Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08801571Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.088128513Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.088223565Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.088282716Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.088300026Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.088306026Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.088351657Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089280246Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089353928Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-tr78b/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089404799Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.08945278Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089515131Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089563072Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089650864Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089705365Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089767876Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089810347Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089882398Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.089928129Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090013331Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090131924Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090254846Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090304737Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090354268Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090419129Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09047187Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090539742Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090587863Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090637604Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090722176Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090761326Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090771466Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090822687Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090912339Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.090989831Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.091049722Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.091149224Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.091353268Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.091663545Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.091741376Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.091854559Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092018492Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092179615Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092241716Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092291627Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092360069Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09240451Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092476941Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092510272Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092583013Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092618844Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092703636Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092791547Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092853599Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09289744Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.092961931Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.093004802Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.093096454Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.093135025Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.093207916Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.093248387Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.093332009Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.093341989Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09339411Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.093549893Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.0938914Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094031633Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094241457Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094438211Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094447371Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-mdrqd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094476172Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-mdrqd/volume-data-source-validator-549c6d54cf-mdrqd.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094561824Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-mdrqd/volume-data-source-validator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094572364Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-mdrqd/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094577054Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-mdrqd/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094598074Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-mdrqd/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094709877Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-mdrqd/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094776218Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-mdrqd/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094795618Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09485438Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094901881Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094916291Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.094943591Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095013683Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095056174Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095123515Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095163166Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095267788Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095313899Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09537228Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095426291Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095470432Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095526483Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095605895Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095685206Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095765438Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095823259Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09586826Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095922431Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.095969352Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096047674Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096106555Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096153516Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096263208Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096309859Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096320619Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09636481Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096443112Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096485663Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096596675Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096670897Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096757539Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09683969Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.096918282Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097038144Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097228718Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097300199Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09734453Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097409232Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097451153Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097499683Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097538494Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097624876Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097714028Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097761589Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09780645Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097861791Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097910192Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097958403Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.097974923Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.098018024Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.098183597Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.098376521Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.098512794Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.098717408Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.098928752Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.098969713Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099031635Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099069495Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099179418Z namespaces/openshift-config-managed/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099328021Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099546805Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099590526Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099668328Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099690218Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.09978709Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099861662Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099937763Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.099983714Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.100018065Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.100067016Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.100111497Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.100215089Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10026893Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.100315301Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.100396062Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.100470974Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.100516945Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.103800962Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.103877664Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.103967225Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104086498Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10418817Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104340703Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104462765Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104528137Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104578298Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104659849Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10470648Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104754581Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104798792Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104845433Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104902194Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.104954515Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105041697Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105132169Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10518373Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105242791Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105301182Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105349323Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105401144Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105411324Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105471336Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10566986Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.105863464Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106002807Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106220381Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106447796Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106489346Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106547178Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106557518Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106599129Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10666736Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106697201Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106765642Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106798133Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106858024Z namespaces/openshift-config-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106869654Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106879645Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106930415Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.106997967Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.107014717Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.107054948Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.107353094Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.107553708Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.107693681Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.107889535Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108253493Z namespaces/openshift-config/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108304454Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108373705Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108417296Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108487777Z namespaces/openshift-config/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108504088Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10860134Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108677241Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108759983Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108813964Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108867055Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108922996Z namespaces/openshift-config/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.108974237Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.109047429Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10909818Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10912094Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.109241223Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.109265663Z namespaces/openshift-config/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.109324334Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.109440497Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.109519428Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.10960329Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.109688732Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.109945597Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.11008335Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110246403Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110293174Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110344435Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110383396Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110420887Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110479188Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110522909Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.11057323Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110623221Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110699772Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110777704Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110819435Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110857286Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110911836Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.110950497Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.111007449Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.111022039Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.11107134Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.111251083Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.111452287Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.111611391Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.111805985Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.111996719Z namespaces/openshift-config/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112035879Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112088011Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112125741Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112287815Z namespaces/openshift-console-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112332676Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112467748Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112527489Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112613371Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112663092Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112750884Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112873787Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.112975039Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113021899Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.11305256Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113106791Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113136062Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113247814Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113296065Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113329976Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113409658Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113511649Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.11352059Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113559921Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113634032Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.113668833Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.114322246Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.114409218Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.114821116Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.114899908Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115033271Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115150763Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115425028Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.11547298Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115550421Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115598282Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115632773Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115717845Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115784916Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.115907409Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116029281Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116114563Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116163924Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116240845Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116290876Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116350587Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116397348Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116444649Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.11648123Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116537151Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116546761Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116600603Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116748755Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.116928229Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117066722Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117312657Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117594533Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117606473Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-pwdzm/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117650654Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-pwdzm/console-operator-584dcb6b54-pwdzm.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117752906Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-pwdzm/console-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117765836Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-pwdzm/console-operator/console-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117771746Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-pwdzm/console-operator/console-operator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.117817727Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-pwdzm/console-operator/console-operator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118304157Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-pwdzm/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118343898Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118380559Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.11843919Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118489221Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118535142Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118545702Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118561492Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118600013Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.118754546Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.119095493Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.119257786Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.119478201Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.119718346Z namespaces/openshift-console/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.119762107Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120081763Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120102204Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120218276Z namespaces/openshift-console/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120240457Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120341089Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120514532Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120775777Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120820578Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120854959Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12092054Z namespaces/openshift-console/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.120966471Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.121047173Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.121099724Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.121145975Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.121261027Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.121319309Z namespaces/openshift-console/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12136876Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.121975082Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.122062134Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.122464462Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.122542373Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.122723167Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12285552Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123044524Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123117055Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123163106Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12333795Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123384901Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123447262Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123487273Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123593385Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123670586Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123727888Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123762058Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123823359Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12385118Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123922441Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.123969702Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.124019583Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.124029344Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.124070975Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.124278799Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.124502983Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.124641196Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12483792Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125046435Z namespaces/openshift-console/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125059685Z namespaces/openshift-console/pods/console-68d5c4bcf9-b6sdd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125110666Z namespaces/openshift-console/pods/console-68d5c4bcf9-b6sdd/console-68d5c4bcf9-b6sdd.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125235738Z namespaces/openshift-console/pods/console-68d5c4bcf9-b6sdd/console/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125249499Z namespaces/openshift-console/pods/console-68d5c4bcf9-b6sdd/console/console/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125254479Z namespaces/openshift-console/pods/console-68d5c4bcf9-b6sdd/console/console/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125282509Z namespaces/openshift-console/pods/console-68d5c4bcf9-b6sdd/console/console/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125382861Z namespaces/openshift-console/pods/console-68d5c4bcf9-b6sdd/console/console/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125450253Z namespaces/openshift-console/pods/console-68d5c4bcf9-b6sdd/console/console/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125490784Z namespaces/openshift-console/pods/downloads-7c97dc4555-s7hbx/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125531794Z namespaces/openshift-console/pods/downloads-7c97dc4555-s7hbx/downloads-7c97dc4555-s7hbx.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125628336Z namespaces/openshift-console/pods/downloads-7c97dc4555-s7hbx/download-server/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125635407Z namespaces/openshift-console/pods/downloads-7c97dc4555-s7hbx/download-server/download-server/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125639507Z namespaces/openshift-console/pods/downloads-7c97dc4555-s7hbx/download-server/download-server/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.125667927Z namespaces/openshift-console/pods/downloads-7c97dc4555-s7hbx/download-server/download-server/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126056325Z namespaces/openshift-console/pods/downloads-7c97dc4555-s7hbx/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126124737Z namespaces/openshift-console/pods/downloads-7c97dc4555-s7hbx/download-server/download-server/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126156727Z namespaces/openshift-console/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126218418Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12628097Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126330421Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126451233Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126483184Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126558055Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126586386Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126641807Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126670897Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126762459Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126837691Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126918513Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.126967014Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127005184Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127060925Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127113046Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127225479Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12727408Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127322651Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127405103Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127473164Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127535975Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127647808Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127726029Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127805741Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127894042Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.127972754Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128094377Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128433824Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128487025Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128531166Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128584207Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128631737Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128697439Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12874524Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128822571Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128906143Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128952754Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.128993415Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.129041166Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.129089947Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.129136178Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.129143448Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.12923297Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.129355302Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.129539046Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.129673209Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.129870323Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130059177Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130103417Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130150628Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13021222Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130303631Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130342912Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130397473Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130428544Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130485015Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130527276Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130611818Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130692469Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130770891Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130878843Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130912194Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.130983185Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131026216Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131111898Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131153789Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131320442Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131400344Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131457015Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131463985Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131520176Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131613068Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131655899Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131769221Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131847943Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.131931495Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132010626Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132090478Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132237371Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132391214Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132638839Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13268609Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132735411Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132779032Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132828323Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132867714Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.132954465Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133029127Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133075118Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133116889Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13318944Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133230511Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133287192Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133295623Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133328363Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133482596Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13366434Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133800373Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.133996777Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134181861Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134247182Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134291273Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134324424Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134419315Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134466416Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134553168Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134580099Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134792813Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134836624Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.134924066Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135002637Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135085819Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13515275Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135214422Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135304673Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135329184Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135423046Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135478567Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135542638Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13563007Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135697121Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135715862Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135840274Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135912156Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.135997287Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136086919Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136160501Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136313604Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136478707Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136553049Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13660205Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136656201Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136695682Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136748663Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136790383Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136873445Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136950237Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.136984867Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137036419Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13709652Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137129761Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137221752Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137257133Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137313074Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137355025Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137404686Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137414766Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137437017Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13759347Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137769243Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.137910757Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13810332Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.138338225Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.138356506Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.138433447Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.138470988Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.13855991Z namespaces/openshift-dns/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.138599351Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.138663452Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.138697633Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.139587981Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.139634241Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.139790175Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.139872917Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.139951368Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.139989969Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14003121Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140092821Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140133972Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140250184Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140307135Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140351746Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140437338Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140508399Z namespaces/openshift-dns/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14055625Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140662013Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140747454Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.140981779Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.141057001Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.141356147Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.141473339Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.141652293Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.141722194Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.141772145Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.141852457Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.141875617Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142129712Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142164243Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142263175Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142345427Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142413808Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142461309Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14251103Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142554001Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142611482Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142650803Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142684854Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142701714Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142750165Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.142892718Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143074231Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143237605Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143437839Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143660214Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143671124Z namespaces/openshift-dns/pods/dns-default-pj6vq/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143723935Z namespaces/openshift-dns/pods/dns-default-pj6vq/dns-default-pj6vq.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143845577Z namespaces/openshift-dns/pods/dns-default-pj6vq/dns/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143853727Z namespaces/openshift-dns/pods/dns-default-pj6vq/dns/dns/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143858247Z namespaces/openshift-dns/pods/dns-default-pj6vq/dns/dns/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.143909208Z namespaces/openshift-dns/pods/dns-default-pj6vq/dns/dns/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144006541Z namespaces/openshift-dns/pods/dns-default-pj6vq/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144081592Z namespaces/openshift-dns/pods/dns-default-pj6vq/dns/dns/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144119653Z namespaces/openshift-dns/pods/dns-default-pj6vq/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144137393Z namespaces/openshift-dns/pods/dns-default-pj6vq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144145343Z namespaces/openshift-dns/pods/dns-default-pj6vq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144201665Z namespaces/openshift-dns/pods/dns-default-pj6vq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144310917Z namespaces/openshift-dns/pods/dns-default-pj6vq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144379828Z namespaces/openshift-dns/pods/dns-default-pj6vq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144409499Z namespaces/openshift-dns/pods/dns-default-t4lwh/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14445444Z namespaces/openshift-dns/pods/dns-default-t4lwh/dns-default-t4lwh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144545581Z namespaces/openshift-dns/pods/dns-default-t4lwh/dns/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144558602Z namespaces/openshift-dns/pods/dns-default-t4lwh/dns/dns/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144568432Z namespaces/openshift-dns/pods/dns-default-t4lwh/dns/dns/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144592653Z namespaces/openshift-dns/pods/dns-default-t4lwh/dns/dns/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144688495Z namespaces/openshift-dns/pods/dns-default-t4lwh/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144765116Z namespaces/openshift-dns/pods/dns-default-t4lwh/dns/dns/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144796107Z namespaces/openshift-dns/pods/dns-default-t4lwh/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144802667Z namespaces/openshift-dns/pods/dns-default-t4lwh/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144808587Z namespaces/openshift-dns/pods/dns-default-t4lwh/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.144855978Z namespaces/openshift-dns/pods/dns-default-t4lwh/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.1449533Z namespaces/openshift-dns/pods/dns-default-t4lwh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145022301Z namespaces/openshift-dns/pods/dns-default-t4lwh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145078323Z namespaces/openshift-dns/pods/dns-default-zfdj9/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145108733Z namespaces/openshift-dns/pods/dns-default-zfdj9/dns-default-zfdj9.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145225826Z namespaces/openshift-dns/pods/dns-default-zfdj9/dns/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145241646Z namespaces/openshift-dns/pods/dns-default-zfdj9/dns/dns/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145251986Z namespaces/openshift-dns/pods/dns-default-zfdj9/dns/dns/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145301147Z namespaces/openshift-dns/pods/dns-default-zfdj9/dns/dns/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145390189Z namespaces/openshift-dns/pods/dns-default-zfdj9/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14545565Z namespaces/openshift-dns/pods/dns-default-zfdj9/dns/dns/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145484771Z namespaces/openshift-dns/pods/dns-default-zfdj9/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145491841Z namespaces/openshift-dns/pods/dns-default-zfdj9/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145498171Z namespaces/openshift-dns/pods/dns-default-zfdj9/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145559922Z namespaces/openshift-dns/pods/dns-default-zfdj9/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145649944Z namespaces/openshift-dns/pods/dns-default-zfdj9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145717025Z namespaces/openshift-dns/pods/dns-default-zfdj9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145758586Z namespaces/openshift-dns/pods/node-resolver-8lrn8/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145794787Z namespaces/openshift-dns/pods/node-resolver-8lrn8/node-resolver-8lrn8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145894859Z namespaces/openshift-dns/pods/node-resolver-8lrn8/dns-node-resolver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.145902169Z namespaces/openshift-dns/pods/node-resolver-8lrn8/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14591136Z namespaces/openshift-dns/pods/node-resolver-8lrn8/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.1459534Z namespaces/openshift-dns/pods/node-resolver-8lrn8/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146041912Z namespaces/openshift-dns/pods/node-resolver-8lrn8/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146111184Z namespaces/openshift-dns/pods/node-resolver-8lrn8/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146181775Z namespaces/openshift-dns/pods/node-resolver-k56kz/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146224286Z namespaces/openshift-dns/pods/node-resolver-k56kz/node-resolver-k56kz.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146319328Z namespaces/openshift-dns/pods/node-resolver-k56kz/dns-node-resolver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146326308Z namespaces/openshift-dns/pods/node-resolver-k56kz/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146330408Z namespaces/openshift-dns/pods/node-resolver-k56kz/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146372119Z namespaces/openshift-dns/pods/node-resolver-k56kz/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146461451Z namespaces/openshift-dns/pods/node-resolver-k56kz/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146527792Z namespaces/openshift-dns/pods/node-resolver-k56kz/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146565433Z namespaces/openshift-dns/pods/node-resolver-w84kh/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146593523Z namespaces/openshift-dns/pods/node-resolver-w84kh/node-resolver-w84kh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146672665Z namespaces/openshift-dns/pods/node-resolver-w84kh/dns-node-resolver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146680545Z namespaces/openshift-dns/pods/node-resolver-w84kh/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146684685Z namespaces/openshift-dns/pods/node-resolver-w84kh/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146719956Z namespaces/openshift-dns/pods/node-resolver-w84kh/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146805908Z namespaces/openshift-dns/pods/node-resolver-w84kh/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146871469Z namespaces/openshift-dns/pods/node-resolver-w84kh/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14690484Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14693168Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.146977201Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147026702Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147109844Z namespaces/openshift-etcd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147148365Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147309588Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147347919Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14741333Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147451271Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147549673Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147627684Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147707216Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147757547Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147796878Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147845929Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14788905Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.147969671Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148021662Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148052383Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148143975Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148243907Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148262927Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14838454Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148457121Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148544823Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148616344Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148696496Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148822699Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.148981762Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149045393Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149086574Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149126935Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149180196Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149240207Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149277938Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14936418Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149440561Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149503802Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149535193Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149589774Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149618115Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149673386Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149684606Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.149735367Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.14988685Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.150069374Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.150226407Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.150550294Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.150745778Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.150784909Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15085248Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15088043Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.150999173Z namespaces/openshift-host-network/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151049954Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151108935Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151146076Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151240908Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151267918Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.1513668Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151442372Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151537994Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151590285Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151623476Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151689017Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151727928Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151810269Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151866481Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151912122Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.151990633Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152072205Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152117046Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152248478Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15232405Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152400101Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152479143Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152559715Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152707458Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15281167Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152859521Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152902202Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.152971053Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153016494Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153077065Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153115816Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153190438Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153239999Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153328451Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153404212Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153463123Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153510814Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153571605Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153623006Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153666397Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153694588Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153752019Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.153921383Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.154140117Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15430236Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.154504634Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.154705358Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.154735029Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15479536Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.154839681Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.154916333Z namespaces/openshift-image-registry/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.154946233Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155024855Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155064686Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155097356Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155150677Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155429293Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155577436Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15578255Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155817071Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155867642Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155898683Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.155955914Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.156055476Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.156118327Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.156150498Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15626678Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.156313001Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.156318311Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.156361742Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.156441044Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.156484575Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.157556657Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.157678999Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.157939704Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158021046Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158263861Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158385823Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158583347Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158641349Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.15869304Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158744051Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158783071Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158833052Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158871833Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.158955515Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159036457Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159104388Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159140699Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159246271Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159316773Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159445055Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159548837Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159616968Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159668469Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159726831Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159764362Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159815712Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159830973Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.159891004Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.160040167Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.160248541Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.160410535Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.160632389Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.160907615Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.160917025Z namespaces/openshift-image-registry/pods/image-registry-d8bcdfb7f-gtf7k/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.160954776Z namespaces/openshift-image-registry/pods/image-registry-d8bcdfb7f-gtf7k/image-registry-d8bcdfb7f-gtf7k.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.161065648Z namespaces/openshift-image-registry/pods/image-registry-d8bcdfb7f-gtf7k/registry/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.161073488Z namespaces/openshift-image-registry/pods/image-registry-d8bcdfb7f-gtf7k/registry/registry/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.161084218Z namespaces/openshift-image-registry/pods/image-registry-d8bcdfb7f-gtf7k/registry/registry/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.161129029Z namespaces/openshift-image-registry/pods/image-registry-d8bcdfb7f-gtf7k/registry/registry/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.16360995Z namespaces/openshift-image-registry/pods/image-registry-d8bcdfb7f-gtf7k/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.163683471Z namespaces/openshift-image-registry/pods/image-registry-d8bcdfb7f-gtf7k/registry/registry/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.163718722Z namespaces/openshift-image-registry/pods/node-ca-72mk7/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.163756743Z namespaces/openshift-image-registry/pods/node-ca-72mk7/node-ca-72mk7.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.163850225Z namespaces/openshift-image-registry/pods/node-ca-72mk7/node-ca/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.163860595Z namespaces/openshift-image-registry/pods/node-ca-72mk7/node-ca/node-ca/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.163865865Z namespaces/openshift-image-registry/pods/node-ca-72mk7/node-ca/node-ca/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.163877495Z namespaces/openshift-image-registry/pods/node-ca-72mk7/node-ca/node-ca/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.16408923Z namespaces/openshift-image-registry/pods/node-ca-72mk7/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164156911Z namespaces/openshift-image-registry/pods/node-ca-72mk7/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164229133Z namespaces/openshift-image-registry/pods/node-ca-ckvrn/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164249933Z namespaces/openshift-image-registry/pods/node-ca-ckvrn/node-ca-ckvrn.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164329044Z namespaces/openshift-image-registry/pods/node-ca-ckvrn/node-ca/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164338345Z namespaces/openshift-image-registry/pods/node-ca-ckvrn/node-ca/node-ca/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164348105Z namespaces/openshift-image-registry/pods/node-ca-ckvrn/node-ca/node-ca/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164404996Z namespaces/openshift-image-registry/pods/node-ca-ckvrn/node-ca/node-ca/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.16460234Z namespaces/openshift-image-registry/pods/node-ca-ckvrn/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164670991Z namespaces/openshift-image-registry/pods/node-ca-ckvrn/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164705052Z namespaces/openshift-image-registry/pods/node-ca-q8nkb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164735923Z namespaces/openshift-image-registry/pods/node-ca-q8nkb/node-ca-q8nkb.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164821014Z namespaces/openshift-image-registry/pods/node-ca-q8nkb/node-ca/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164828145Z namespaces/openshift-image-registry/pods/node-ca-q8nkb/node-ca/node-ca/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164837065Z namespaces/openshift-image-registry/pods/node-ca-q8nkb/node-ca/node-ca/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.164909466Z namespaces/openshift-image-registry/pods/node-ca-q8nkb/node-ca/node-ca/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.16511372Z namespaces/openshift-image-registry/pods/node-ca-q8nkb/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165182762Z namespaces/openshift-image-registry/pods/node-ca-q8nkb/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165245683Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165283474Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165355955Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165380386Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165436807Z namespaces/openshift-infra/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165444047Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165448007Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165486628Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.165646441Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166027389Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166188932Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166429587Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166721063Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166769934Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166841095Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166870846Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166947798Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.166983118Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167127661Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167242054Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167322255Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167374246Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167416417Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167484149Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.1675473Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167637222Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167682433Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167711303Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167804465Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167897097Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.167934988Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.16805288Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.168141692Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.168330886Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.168408058Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.168573481Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.168695813Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.168867087Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.168920198Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.168962309Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.16901573Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169063821Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169130302Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169162803Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169280905Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169352027Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169397808Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169438759Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.16948944Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.16953217Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169588551Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169597932Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169649633Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169801806Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.169981139Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.170130343Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.170352747Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.170825717Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.170835277Z namespaces/openshift-ingress-canary/pods/ingress-canary-dmr84/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.170874078Z namespaces/openshift-ingress-canary/pods/ingress-canary-dmr84/ingress-canary-dmr84.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17097866Z namespaces/openshift-ingress-canary/pods/ingress-canary-dmr84/serve-healthcheck-canary/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17098769Z namespaces/openshift-ingress-canary/pods/ingress-canary-dmr84/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17099232Z namespaces/openshift-ingress-canary/pods/ingress-canary-dmr84/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171030161Z namespaces/openshift-ingress-canary/pods/ingress-canary-dmr84/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171133493Z namespaces/openshift-ingress-canary/pods/ingress-canary-dmr84/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171303026Z namespaces/openshift-ingress-canary/pods/ingress-canary-dmr84/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171357678Z namespaces/openshift-ingress-canary/pods/ingress-canary-dvrgq/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171386828Z namespaces/openshift-ingress-canary/pods/ingress-canary-dvrgq/ingress-canary-dvrgq.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17147239Z namespaces/openshift-ingress-canary/pods/ingress-canary-dvrgq/serve-healthcheck-canary/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17147954Z namespaces/openshift-ingress-canary/pods/ingress-canary-dvrgq/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17148589Z namespaces/openshift-ingress-canary/pods/ingress-canary-dvrgq/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171563972Z namespaces/openshift-ingress-canary/pods/ingress-canary-dvrgq/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171669354Z namespaces/openshift-ingress-canary/pods/ingress-canary-dvrgq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171737975Z namespaces/openshift-ingress-canary/pods/ingress-canary-dvrgq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171754116Z namespaces/openshift-ingress-canary/pods/ingress-canary-f7c9g/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171807717Z namespaces/openshift-ingress-canary/pods/ingress-canary-f7c9g/ingress-canary-f7c9g.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171891698Z namespaces/openshift-ingress-canary/pods/ingress-canary-f7c9g/serve-healthcheck-canary/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171899589Z namespaces/openshift-ingress-canary/pods/ingress-canary-f7c9g/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.171903799Z namespaces/openshift-ingress-canary/pods/ingress-canary-f7c9g/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17194988Z namespaces/openshift-ingress-canary/pods/ingress-canary-f7c9g/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172049802Z namespaces/openshift-ingress-canary/pods/ingress-canary-f7c9g/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172124593Z namespaces/openshift-ingress-canary/pods/ingress-canary-f7c9g/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172183195Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172225805Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172264986Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172315797Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17244965Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172490391Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172581523Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172624943Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172696715Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172738366Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.172858818Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17294644Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173029942Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173083313Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173133634Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173215985Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173249836Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173467341Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173523512Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173571633Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173646514Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173723686Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.173752456Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.174667855Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.174753517Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.1749091Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.174992421Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175070513Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175228176Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17541289Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175479671Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175549243Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175617814Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175653955Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175718326Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175726107Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175765887Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175839729Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17588056Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.175957271Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176040903Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176096754Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176143615Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176225517Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176286568Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17636436Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.1763954Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176467262Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176502872Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176551413Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176559184Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176597294Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176684256Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176691856Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176745607Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.176906821Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177098195Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177251188Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177454542Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177663486Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177701317Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177757708Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177804009Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177910681Z namespaces/openshift-ingress/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.177959192Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178013773Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178059554Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178130825Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178163446Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178275168Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178505233Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178703637Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178747548Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178780359Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17886066Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178892821Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.178983023Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179045034Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179091005Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179183747Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179598595Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179606246Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179662077Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179762019Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.17984976Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179954993Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.179989063Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.180445393Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.180544445Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.180757509Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.180831811Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181081676Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181228088Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181423952Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181538335Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181586546Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181666337Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181674668Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181696308Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.18180523Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181844161Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181916613Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.181946173Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182048265Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182142567Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182222379Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.18225042Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182349351Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182377192Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182469784Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182498034Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182563726Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182609467Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182663098Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182692709Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182715959Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.182919763Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183097567Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.18326109Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183449504Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183670498Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183681809Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183717449Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/istiod-openshift-gateway-5677ccb49b-8bfqq.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183845112Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183852222Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183858592Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.183902743Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.193016519Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.19308355Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-8bfqq/discovery/discovery/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.193147251Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.193194652Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/openshift-ai-inference-openshift-default-8699b59bbc-spfsb.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.193342315Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.193356126Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.193361696Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.193398807Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.194469248Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-spfsb/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.19455689Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.194602871Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router-default-b46967c9f-ptz2d.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.194721083Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.194730274Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.194740644Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.194781725Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.194886007Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.194956088Z namespaces/openshift-ingress/pods/router-default-b46967c9f-ptz2d/router/router/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195007259Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.19504425Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195112521Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195147312Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195328176Z namespaces/openshift-insights/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195367827Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195441688Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195475239Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195568231Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195612741Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195733834Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195849016Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195950528Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.195970209Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.19603563Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196122152Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196179843Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196252074Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196299275Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196380927Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196434518Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196460279Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196546871Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196628672Z namespaces/openshift-insights/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.196674033Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.197604272Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.197726395Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.19800682Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.198137513Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.198452149Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.198539351Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.198696284Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.198763706Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.198812037Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.198881848Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.198926789Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199015131Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199075912Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199126963Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199215305Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199268206Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199334637Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199375158Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.19944751Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199488551Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199592562Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199664644Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199722185Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199769726Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199841218Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199856088Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.19993783Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.199986421Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200042892Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200081323Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200128493Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200138074Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200216045Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200392959Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200620483Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200769537Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.200980401Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.201238576Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.201249836Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-jvqln/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.201295357Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-jvqln/insights-operator-5885fdb759-jvqln.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.201394119Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-jvqln/insights-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20140387Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-jvqln/insights-operator/insights-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20140914Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-jvqln/insights-operator/insights-operator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.201459981Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-jvqln/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202081423Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-jvqln/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20242059Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202455121Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/insights-runtime-extractor-9jp2v.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202581674Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202590214Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/exporter/exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202595064Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/exporter/exporter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202646795Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/exporter/exporter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202747997Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202818208Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/exporter/exporter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202847149Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/extractor/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202851609Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/extractor/extractor/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.202857229Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/extractor/extractor/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20290846Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/extractor/extractor/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203050753Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203162475Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/extractor/extractor/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203244467Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203261017Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203265147Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203339619Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203467092Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203590064Z namespaces/openshift-insights/pods/insights-runtime-extractor-9jp2v/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203644315Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203687676Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/insights-runtime-extractor-cz9hj.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203818639Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203825889Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/exporter/exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.203833949Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/exporter/exporter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20390358Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/exporter/exporter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204030193Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204137865Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/exporter/exporter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204318279Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/extractor/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204333489Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/extractor/extractor/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204338559Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/extractor/extractor/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20438688Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/extractor/extractor/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204488922Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204563254Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/extractor/extractor/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204589734Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204780958Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204788348Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204830289Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.204935721Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205002063Z namespaces/openshift-insights/pods/insights-runtime-extractor-cz9hj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205060964Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205106095Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/insights-runtime-extractor-dd9n4.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205222597Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205233917Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/exporter/exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205237897Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/exporter/exporter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205281989Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/exporter/exporter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20537Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205438452Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/exporter/exporter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205481362Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/extractor/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205489893Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/extractor/extractor/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205494063Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/extractor/extractor/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205533963Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/extractor/extractor/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205631616Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.205966832Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/extractor/extractor/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206006983Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206015203Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206019153Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206064014Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206159526Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206262028Z namespaces/openshift-insights/pods/insights-runtime-extractor-dd9n4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206294619Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20634206Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206400081Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206445432Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206475003Z namespaces/openshift-keda/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206479683Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206485633Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206542344Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206637886Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206677737Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206686357Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206729738Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206784429Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20682995Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206915582Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.206998453Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.207058045Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.207065125Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.207107186Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.207283069Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.207474793Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.207719008Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.207926072Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208071185Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208281919Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208329571Z namespaces/openshift-keda/operators.coreos.com/installplans/install-hlp2f.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208449803Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208490794Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208554985Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208595246Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208663347Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208706528Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20879339Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208842551Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208905882Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.208934173Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209005874Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209048185Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209132027Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209241459Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209321321Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209373742Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209423053Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209472174Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209515865Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209594176Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209647067Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209690808Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.20976917Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209826411Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209870452Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.209976744Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210062696Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210140887Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21025227Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210333311Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210451244Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210624117Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210674568Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210715769Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21075111Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210808421Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210843852Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210896863Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.210978865Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211062376Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211112597Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211149968Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21123901Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211272851Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211324312Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211332042Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211382133Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211545676Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21172116Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.211864292Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212058547Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212274501Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212318442Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212371153Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212414564Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212479785Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212522426Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212587167Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212625348Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212673719Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21272676Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212812432Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212892893Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.212971505Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213032526Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213045527Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213108028Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213137509Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213263031Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213313992Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213363173Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213441195Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213485325Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213519996Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213628008Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21370832Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213787242Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213865553Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.213943945Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214068788Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214256241Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214318522Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214366314Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214415795Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214457695Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214507526Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214555907Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214629639Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214711281Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214763782Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214807092Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214874284Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.214918565Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215015337Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215074888Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215107699Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21518795Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215240051Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215290962Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215299722Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215341393Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215507057Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215691941Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.215831393Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216027567Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216243412Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216274422Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216333823Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216362104Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216445706Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216485317Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216550978Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216578328Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21664269Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21667205Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216764182Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216839624Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216920815Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216965006Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.216993597Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217056008Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217099109Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217212821Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217273313Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217313123Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217390995Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217446656Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217490677Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217599839Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217682321Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217761103Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217842384Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.217916256Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218042078Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218237282Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218300034Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218346824Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218394605Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218423636Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218484877Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218512428Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21860246Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218681421Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218732882Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218760363Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218828864Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218877995Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218928736Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.218956887Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.219013788Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.219023048Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.219071359Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.219409476Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.21960642Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.219743983Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.219935467Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220142591Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220185932Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220280584Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220317015Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220435757Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220496458Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22058981Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220630131Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220718973Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220735193Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.220841365Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22109146Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221107301Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221113551Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221153522Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221242503Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221287935Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221377916Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221425727Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221470218Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22154902Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221595601Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221603311Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221644742Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221714973Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221755634Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221862566Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.221937028Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.222405537Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.222483349Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22256176Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.222690843Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.222855276Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.222909707Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.222950628Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.222983779Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22304229Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223093761Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223145522Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223244474Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223321756Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223371897Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223410087Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223461849Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223497109Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223562621Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223572771Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223582641Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223749274Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.223933098Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224067831Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224294666Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22449348Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22452862Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224581242Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224614532Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224683884Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224726174Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224787106Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224828647Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224879048Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.224931509Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22500153Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225085152Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225161013Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225232825Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225260775Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225318357Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225355567Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225433579Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22548303Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225512941Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225589172Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225652963Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225692914Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225800196Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225873458Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.225955359Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226035231Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226117343Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226259486Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226423459Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22647622Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226522281Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226570352Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226612433Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226664794Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226707475Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226788667Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226871098Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.226919049Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.2269535Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.227007981Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.227048262Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.227095573Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.227104423Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.227146534Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.227409049Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.227609063Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.227743686Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22793746Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228131224Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228185615Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228226566Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228277377Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228342458Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228384949Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22844221Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228477231Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228532732Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228571323Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228652374Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228728576Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228808268Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228861318Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228895829Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228952011Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.228994401Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229070623Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229121924Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229156135Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229263977Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229317098Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229362139Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229459591Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229533822Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229614994Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229696746Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.229775687Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.22989511Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230056273Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230104324Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230145075Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230222846Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230271687Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230323328Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230363019Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230444651Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230528503Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230576893Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230616054Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230663625Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230708966Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230752217Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230760397Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230801298Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.230953751Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23140044Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.231548173Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.231747768Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.231930641Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.231971162Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232028463Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232069344Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232141765Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232192906Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232272208Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232306229Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23237924Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232408871Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232501643Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232626095Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232725257Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232760118Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232814129Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23286445Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232902171Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.232974302Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233025763Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233072544Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233150716Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233230598Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233242738Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233274788Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23335013Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233390401Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233501953Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233587375Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.233754038Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23383086Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234008323Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234147926Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234355541Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234413252Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234452463Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234508393Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234536314Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234597815Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234626806Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234721948Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23480647Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234859221Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234888801Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234951923Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.234986253Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.235038064Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.235047755Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.235086395Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.235340041Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.235548295Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.235687238Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.235885372Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236060415Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236067875Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236110556Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236258329Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t/kube-storage-version-migrator-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236269809Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23627946Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236323681Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236441223Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236566686Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236597906Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236648207Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236692598Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236748689Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23677933Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236850831Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.236883752Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237062726Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237092886Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237186198Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237317651Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237416983Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237451913Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237498265Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237563496Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237610557Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237691169Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23774579Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23778013Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237862012Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237915873Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.237963064Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238047656Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238133338Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23827038Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238349422Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238462584Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238577996Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.23874569Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238802511Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238848992Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238900563Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238935744Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.238993835Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239025596Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239109177Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239193279Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239263661Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239297711Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239348162Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239394833Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239451004Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239460825Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239505326Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239664069Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239843362Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.239978495Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240183589Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240399484Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240411674Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240446045Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/migrator-6865c9967-wc267.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240530516Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/graceful-termination/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240539587Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/graceful-termination/graceful-termination/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240543667Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240587817Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240679239Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240748321Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240778241Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/migrator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240786012Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/migrator/migrator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240790592Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/migrator/migrator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240831092Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/migrator/migrator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.240932555Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241006306Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wc267/migrator/migrator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241033986Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241075987Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241125138Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24118554Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241239421Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241291502Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241365123Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241405614Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241469715Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241506976Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241588048Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241737661Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241864673Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241903984Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241944105Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.241993026Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242043367Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242116359Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242164489Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242221211Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242310183Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242356793Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242372194Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242401414Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242485896Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242569508Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242615509Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242731031Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.242823373Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.243359654Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.243467216Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.243834774Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.243984157Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244251542Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244329054Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244390055Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244484977Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244537878Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244614719Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24466618Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244773173Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244858744Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244936886Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.244985387Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.245077959Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24511581Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.245235282Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.245291403Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.245368205Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.245377625Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.245447036Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24563496Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.245847565Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.246012068Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.246316754Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.246553489Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.246786334Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.246842045Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-njj7d.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.246939347Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.246991338Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.247071749Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24711886Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.247222723Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.247279734Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.247377446Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.247384256Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-h8v7t/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.247438357Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-h8v7t/lws-controller-manager-67d9b7fdbd-h8v7t.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24756363Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-h8v7t/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24757101Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-h8v7t/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24758143Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-h8v7t/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.247651871Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-h8v7t/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248052949Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-h8v7t/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248142031Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-h8v7t/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248226323Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-hckgz/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248278714Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-hckgz/lws-controller-manager-67d9b7fdbd-hckgz.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248378316Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-hckgz/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248385236Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-hckgz/manager/manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248417057Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-hckgz/manager/manager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248472188Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-hckgz/manager/manager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248594791Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-hckgz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248681902Z namespaces/openshift-lws-operator/pods/lws-controller-manager-67d9b7fdbd-hckgz/manager/manager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248744574Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-zv4dc/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248807105Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-zv4dc/openshift-lws-operator-fd8ccff4c-zv4dc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248906727Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-zv4dc/openshift-lws-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248913767Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-zv4dc/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.248937438Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-zv4dc/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249003419Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-zv4dc/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249232624Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-zv4dc/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249329755Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-zv4dc/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249378047Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249431577Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249492689Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.24954746Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249604311Z namespaces/openshift-machine-api/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249611391Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249615321Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249667582Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.249982469Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.250215604Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.250385007Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.250625682Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.250848446Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.250858727Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.250869277Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.250944018Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.251071421Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.251182933Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.251317636Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.251368217Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.251567061Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.251776475Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.251936989Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252184134Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252443229Z namespaces/openshift-marketplace/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252450309Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252457169Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252530571Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252599122Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252606992Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252661083Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252775476Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.252881708Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.25298901Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.253082222Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.253186844Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.253392248Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.253597822Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.253770296Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.25399887Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.25448137Z namespaces/openshift-monitoring/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.254508521Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.254625983Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.254667164Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.254728515Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.254776426Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.25498091Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.25542893Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.255809718Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256037722Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256070893Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256144274Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256163405Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256290477Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256352478Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256370849Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256476191Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256499452Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.256563493Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.260192957Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.26037216Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.261309189Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.261392301Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.262464903Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.262593116Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263079406Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263265549Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.26329897Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263466593Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263522804Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263554645Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263608426Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263693248Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.26376866Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263827491Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.263874842Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264002974Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264031395Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264126407Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264183698Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.26428653Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264329011Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264431303Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264625887Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264711809Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.264965334Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.265101127Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.26524907Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.265337411Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.265444383Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.265532365Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.265607327Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.265641367Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.2657394Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.265824351Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.265918633Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266019445Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266102987Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266279291Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266372293Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266464694Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266563386Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266651318Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.26674456Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266808391Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266851872Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266902213Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266910093Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.266944944Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.267131118Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.267421414Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.267607228Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.267851273Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268092168Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268133258Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268337442Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268360563Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268452165Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268722661Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.26873049Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268734411Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268807772Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.268951345Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269057347Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269099958Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269113668Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269120769Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.26918446Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269317442Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269429865Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269468066Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269477026Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269487866Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269538437Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269641929Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269719141Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269760531Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269772022Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269779582Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269817723Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.269946565Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270018577Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270211301Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270223591Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270229581Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270244281Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270259202Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270338513Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270402735Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270414905Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270420335Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270477306Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270602229Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270708731Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270763152Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270773252Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270783332Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270837544Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.270935195Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.271005597Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.271048978Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-4pw8b/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.271093179Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-4pw8b/cluster-monitoring-operator-548b489cb8-4pw8b.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.271228581Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-4pw8b/cluster-monitoring-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.271244862Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-4pw8b/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.271257782Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-4pw8b/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.271314133Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-4pw8b/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.272096229Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-4pw8b/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.27359372Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273651251Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-state-metrics-56d9f944b4-pxsf7.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273748313Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-main/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273760233Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273767623Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273803524Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273898846Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273971167Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273987528Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-self/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.273999548Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274008778Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274052989Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274146681Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274240263Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274260003Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-state-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274268803Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274278983Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274326475Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274524679Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.27459266Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-pxsf7/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274633441Z namespaces/openshift-monitoring/pods/metrics-server-5f77db5df6-9w52q/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274672031Z namespaces/openshift-monitoring/pods/metrics-server-5f77db5df6-9w52q/metrics-server-5f77db5df6-9w52q.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274765924Z namespaces/openshift-monitoring/pods/metrics-server-5f77db5df6-9w52q/metrics-server/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274780674Z namespaces/openshift-monitoring/pods/metrics-server-5f77db5df6-9w52q/metrics-server/metrics-server/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274786964Z namespaces/openshift-monitoring/pods/metrics-server-5f77db5df6-9w52q/metrics-server/metrics-server/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274826315Z namespaces/openshift-monitoring/pods/metrics-server-5f77db5df6-9w52q/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.274952177Z namespaces/openshift-monitoring/pods/metrics-server-5f77db5df6-9w52q/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275056329Z namespaces/openshift-monitoring/pods/metrics-server-5f77db5df6-9w52q/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.27511122Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-bcrmc/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275164372Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-bcrmc/monitoring-plugin-68b7fc9574-bcrmc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275303304Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-bcrmc/monitoring-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275314145Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-bcrmc/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275324405Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-bcrmc/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275372966Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-bcrmc/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275507019Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-bcrmc/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275614221Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-bcrmc/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275680702Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275724313Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/node-exporter-ct2bc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275864466Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/init-textfile/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275874836Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/init-textfile/init-textfile/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275886516Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/init-textfile/init-textfile/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.275945048Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.27605406Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276132861Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276192173Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276207083Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276211643Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276292625Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276424307Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276521679Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276581121Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/node-exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276595161Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/node-exporter/node-exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276600931Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/node-exporter/node-exporter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276674912Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276816975Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276890297Z namespaces/openshift-monitoring/pods/node-exporter-ct2bc/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.276950518Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277007749Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/node-exporter-kcv6b.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277298405Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/init-textfile/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277310065Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/init-textfile/init-textfile/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277314065Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/init-textfile/init-textfile/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277362576Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277465008Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277570611Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277625562Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277635382Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277641102Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277697133Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277828156Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277928888Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277978599Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/node-exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277985649Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/node-exporter/node-exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.277989629Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/node-exporter/node-exporter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.27805495Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278208723Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278302096Z namespaces/openshift-monitoring/pods/node-exporter-kcv6b/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278349906Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278425148Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/node-exporter-p64pw.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.27854198Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/init-textfile/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278549141Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/init-textfile/init-textfile/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278558711Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/init-textfile/init-textfile/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278621792Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278729864Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278811116Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278846027Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278854147Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278863547Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.278934148Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279061501Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279188334Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279240684Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/node-exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279250405Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/node-exporter/node-exporter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279256645Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/node-exporter/node-exporter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279330206Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.27947898Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279585311Z namespaces/openshift-monitoring/pods/node-exporter-p64pw/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279632472Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279667753Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/openshift-state-metrics-b8f49f5c6-xsxnf.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279851397Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-main/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279861167Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279870387Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.279927958Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280044261Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280114412Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280152843Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-self/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280163883Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280192304Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280232855Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280327827Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280401288Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280435869Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/openshift-state-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280446939Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280456499Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28051647Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280628223Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280693854Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-xsxnf/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280757006Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.280826697Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281016891Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281024531Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281034891Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281080592Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281207175Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281279696Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281322357Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281330597Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281334527Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281378218Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28147686Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281542371Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281586962Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281597673Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281603713Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281613853Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281714175Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281780796Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281817057Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281827217Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281833448Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.281863118Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28195528Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282024091Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282067532Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282077602Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282083202Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282092653Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282209035Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282279326Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282320857Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282329918Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282333858Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.282358348Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283049752Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283120744Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283154644Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283161435Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283182345Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283272707Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283379739Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28344797Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283505022Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283550712Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/prometheus-operator-84d69f794d-79npr.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283653745Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283660945Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283664925Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283719616Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283823888Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283895289Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28393583Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/prometheus-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28394237Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28394636Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.283991311Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284333728Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28440456Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-79npr/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28444285Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-jcnth/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284477301Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-jcnth/prometheus-operator-admission-webhook-7f8d78f54b-jcnth.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284617964Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-jcnth/prometheus-operator-admission-webhook/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284625704Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-jcnth/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284633684Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-jcnth/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284674835Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-jcnth/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284779967Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-jcnth/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28488981Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-jcnth/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284939821Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.284992222Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/telemeter-client-d5dd789bc-gkd27.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285137255Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285145315Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285149975Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285222756Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285317788Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28538516Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285447541Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/reload/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285454741Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/reload/reload/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285460611Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/reload/reload/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285528623Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/reload/reload/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285644895Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285721667Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/reload/reload/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285767557Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/telemeter-client/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285777348Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/telemeter-client/telemeter-client/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285783368Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285828279Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.285922441Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286026603Z namespaces/openshift-monitoring/pods/telemeter-client-d5dd789bc-gkd27/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286075584Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286152505Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/thanos-querier-7f67999996-4qfvx.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286464652Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286478152Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286484612Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286516853Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286627135Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286732317Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286794949Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-rules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286802218Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286806239Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.2868756Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.286971722Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287046563Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287095775Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-web/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287105935Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287115145Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287180086Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287321009Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287390021Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287431982Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287440352Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287448832Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287504183Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287603965Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287707587Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287740958Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/prom-label-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287751018Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287758238Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287798869Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287894671Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.287964572Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288010873Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/thanos-query/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288018063Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/thanos-query/thanos-query/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288022363Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/thanos-query/thanos-query/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288065794Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288189127Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288295929Z namespaces/openshift-monitoring/pods/thanos-querier-7f67999996-4qfvx/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28835455Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288404011Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288451842Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288486133Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288615466Z namespaces/openshift-multus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288652826Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288764679Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28882608Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288876541Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.288924482Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.289195967Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.28930258Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.289385781Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.289435022Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.289479543Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.289633596Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.2898009Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.289879471Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.289925622Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.289971013Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.290016894Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.290096146Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.290145337Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.290209148Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29033585Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.290449493Z namespaces/openshift-multus/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.290491224Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.290641757Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.290727598Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29130972Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.291444723Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29225521Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.292341121Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.292476624Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.292532965Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.292581476Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.292673548Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29276539Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.292849822Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.292947704Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.292982804Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293081786Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293164198Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29327949Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293323761Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293381023Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293435954Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293478934Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293533866Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293566266Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293625117Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293660098Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29376385Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293832502Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293889553Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293937754Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.293988715Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.294027946Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.294114118Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.294161308Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29424161Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.294260941Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.294307931Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.294569567Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.294828922Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.294984015Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.295239601Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.295501716Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.295519546Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.295621368Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/multus-additional-cni-plugins-bpmlj.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29574028Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/bond-cni-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.295757751Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.295794422Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.295870013Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.295962615Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296048207Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296067847Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296075697Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/cni-plugins/cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296083847Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296156079Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296274291Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296352533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296374753Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/egress-router-binary-copy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296385374Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296393884Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296437295Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296522916Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296592908Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296624598Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296629829Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296652989Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29670209Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296777652Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296849243Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296877674Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/routeoverride-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296882394Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296886164Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.296927475Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297018026Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297092338Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297136199Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni-bincopy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297147689Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297153759Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297161229Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297292622Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297376504Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297439305Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297446935Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297450915Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297510017Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297615099Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29768457Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bpmlj/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29770136Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297771252Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/multus-additional-cni-plugins-w7blt.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297900815Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/bond-cni-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297908805Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297912735Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.297949856Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298074728Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29814588Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298196311Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298205411Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/cni-plugins/cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298209331Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298248992Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298341904Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298411885Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298444956Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/egress-router-binary-copy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298449926Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298454306Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298499717Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298616309Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298688801Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298720261Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298724981Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298728721Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298773242Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298850674Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298923076Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298961476Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/routeoverride-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298966456Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.298970466Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299026418Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29913948Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299302343Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299324734Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni-bincopy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299336794Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299345544Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299388205Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299482967Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299581379Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29963402Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29964225Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.29965113Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299694281Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299787093Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299855375Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-w7blt/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299910085Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.299956937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/multus-additional-cni-plugins-xdrd5.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300093139Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/bond-cni-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30010296Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30010832Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300188561Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300288833Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300357625Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300420466Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300427916Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/cni-plugins/cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300432786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300470467Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300569789Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30063785Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300653931Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/egress-router-binary-copy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300665081Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300669241Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300718762Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300846315Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300910156Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300945307Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300961757Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300966817Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.300991458Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301059289Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301142181Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301158721Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/routeoverride-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301162851Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301200142Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301241873Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301340745Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301423006Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301460247Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni-bincopy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301465737Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301470147Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301519468Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30161252Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301675792Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301707992Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301719562Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301725002Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301744143Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301852055Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301942177Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xdrd5/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.301966837Z namespaces/openshift-multus/pods/multus-gdzng/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.302022239Z namespaces/openshift-multus/pods/multus-gdzng/multus-gdzng.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30211573Z namespaces/openshift-multus/pods/multus-gdzng/kube-multus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.302126721Z namespaces/openshift-multus/pods/multus-gdzng/kube-multus/kube-multus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.302132291Z namespaces/openshift-multus/pods/multus-gdzng/kube-multus/kube-multus/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.302195682Z namespaces/openshift-multus/pods/multus-gdzng/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30307528Z namespaces/openshift-multus/pods/multus-gdzng/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.303197323Z namespaces/openshift-multus/pods/multus-gdzng/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.303255264Z namespaces/openshift-multus/pods/multus-njw42/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.303312585Z namespaces/openshift-multus/pods/multus-njw42/multus-njw42.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.303396216Z namespaces/openshift-multus/pods/multus-njw42/kube-multus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.303406827Z namespaces/openshift-multus/pods/multus-njw42/kube-multus/kube-multus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.303411187Z namespaces/openshift-multus/pods/multus-njw42/kube-multus/kube-multus/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.303436197Z namespaces/openshift-multus/pods/multus-njw42/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.304321585Z namespaces/openshift-multus/pods/multus-njw42/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.304388277Z namespaces/openshift-multus/pods/multus-njw42/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.304425328Z namespaces/openshift-multus/pods/multus-tr5lq/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.304458338Z namespaces/openshift-multus/pods/multus-tr5lq/multus-tr5lq.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30454956Z namespaces/openshift-multus/pods/multus-tr5lq/kube-multus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30455811Z namespaces/openshift-multus/pods/multus-tr5lq/kube-multus/kube-multus/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.304577031Z namespaces/openshift-multus/pods/multus-tr5lq/kube-multus/kube-multus/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.304616701Z namespaces/openshift-multus/pods/multus-tr5lq/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305470569Z namespaces/openshift-multus/pods/multus-tr5lq/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30553123Z namespaces/openshift-multus/pods/multus-tr5lq/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305563681Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305615052Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/network-metrics-daemon-29lnk.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305698534Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305708124Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305716374Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305754515Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305847086Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305915268Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305950199Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/network-metrics-daemon/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305958949Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.305964579Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30601629Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306214494Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306275785Z namespaces/openshift-multus/pods/network-metrics-daemon-29lnk/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306315926Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306357387Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/network-metrics-daemon-mzhnc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306439138Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306448979Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306454159Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306480449Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306565821Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306645093Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306663963Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/network-metrics-daemon/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306669143Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306676453Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306725054Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.306915948Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30698588Z namespaces/openshift-multus/pods/network-metrics-daemon-mzhnc/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307026301Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307062701Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/network-metrics-daemon-vtptl.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307148563Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307158723Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307189294Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307290986Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307376368Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307450839Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307469519Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/network-metrics-daemon/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30747819Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30750593Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307568281Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307743265Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307810486Z namespaces/openshift-multus/pods/network-metrics-daemon-vtptl/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307843787Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307887068Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307942299Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307951489Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.307964589Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308075112Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308095442Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308153033Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308284976Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308353697Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308411278Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30846664Z namespaces/openshift-must-gather-r8n2x/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30847678Z namespaces/openshift-must-gather-r8n2x/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30848206Z namespaces/openshift-must-gather-r8n2x/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308524581Z namespaces/openshift-must-gather-r8n2x/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308686594Z namespaces/openshift-must-gather-r8n2x/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.308870788Z namespaces/openshift-must-gather-r8n2x/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309036681Z namespaces/openshift-must-gather-r8n2x/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309260406Z namespaces/openshift-must-gather-r8n2x/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.30944961Z namespaces/openshift-network-console/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309510241Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309583853Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309621493Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309670604Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309707035Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309792277Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.309906489Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310009281Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310068662Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310134164Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310236206Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310276657Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310308707Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310351838Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31043123Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310472Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310510431Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310588153Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310628544Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310695625Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.310974521Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311059892Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311189395Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311310878Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311487621Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311623694Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311744257Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311797938Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311831998Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311891949Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31192607Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.311995462Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312056453Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312140535Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312217236Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312260587Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312303218Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31239004Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312464671Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312510092Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312541863Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312594834Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312648755Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312713206Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312721077Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312778898Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.312939691Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.313124135Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.313281718Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.313544033Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.313744527Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.313752698Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-k5llc/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.313789468Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-k5llc/networking-console-plugin-74c9f884b7-k5llc.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31387546Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-k5llc/networking-console-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31388502Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-k5llc/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3138901Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-k5llc/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.313905811Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-k5llc/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314008603Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-k5llc/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314101105Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-k5llc/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314114555Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314186016Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314239077Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314284028Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314321499Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31438019Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314428811Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314472772Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314515783Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314567624Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314667856Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314767848Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3148623Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314902671Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.314952292Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315010123Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315054564Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315113195Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315183467Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315315049Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31535192Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315436062Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315477982Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315536054Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315616435Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315664516Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315712987Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315808489Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.315900601Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.316106955Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.316212648Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.316451882Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.316570995Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.316711478Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.316789259Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31683372Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.316894811Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.316914292Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317020094Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317066315Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317134706Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317239718Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317282999Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317364051Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317395132Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317490254Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317566895Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317615376Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317649067Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317704738Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317757459Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31782743Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317878411Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317954913Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.317962663Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.318016564Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.318332411Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.318546355Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.318710448Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.318947083Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319149577Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319160488Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-8wzwg/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319216029Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-8wzwg/network-check-source-7d88cb55d8-8wzwg.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31930366Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-8wzwg/check-endpoints/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319313381Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-8wzwg/check-endpoints/check-endpoints/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319318201Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-8wzwg/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319367832Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-8wzwg/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319544885Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-8wzwg/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319614167Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-8wzwg/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319645898Z namespaces/openshift-network-diagnostics/pods/network-check-target-4vhkj/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319677418Z namespaces/openshift-network-diagnostics/pods/network-check-target-4vhkj/network-check-target-4vhkj.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31976443Z namespaces/openshift-network-diagnostics/pods/network-check-target-4vhkj/network-check-target-container/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.31977287Z namespaces/openshift-network-diagnostics/pods/network-check-target-4vhkj/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3197772Z namespaces/openshift-network-diagnostics/pods/network-check-target-4vhkj/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319814161Z namespaces/openshift-network-diagnostics/pods/network-check-target-4vhkj/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319900343Z namespaces/openshift-network-diagnostics/pods/network-check-target-4vhkj/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.319997085Z namespaces/openshift-network-diagnostics/pods/network-check-target-4vhkj/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320045326Z namespaces/openshift-network-diagnostics/pods/network-check-target-7qfvm/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320092287Z namespaces/openshift-network-diagnostics/pods/network-check-target-7qfvm/network-check-target-7qfvm.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320160358Z namespaces/openshift-network-diagnostics/pods/network-check-target-7qfvm/network-check-target-container/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320187469Z namespaces/openshift-network-diagnostics/pods/network-check-target-7qfvm/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320195489Z namespaces/openshift-network-diagnostics/pods/network-check-target-7qfvm/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32025363Z namespaces/openshift-network-diagnostics/pods/network-check-target-7qfvm/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320381653Z namespaces/openshift-network-diagnostics/pods/network-check-target-7qfvm/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320485264Z namespaces/openshift-network-diagnostics/pods/network-check-target-7qfvm/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320539826Z namespaces/openshift-network-diagnostics/pods/network-check-target-fncdg/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320608417Z namespaces/openshift-network-diagnostics/pods/network-check-target-fncdg/network-check-target-fncdg.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320696309Z namespaces/openshift-network-diagnostics/pods/network-check-target-fncdg/network-check-target-container/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320704789Z namespaces/openshift-network-diagnostics/pods/network-check-target-fncdg/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320713399Z namespaces/openshift-network-diagnostics/pods/network-check-target-fncdg/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32075752Z namespaces/openshift-network-diagnostics/pods/network-check-target-fncdg/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320846812Z namespaces/openshift-network-diagnostics/pods/network-check-target-fncdg/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320922883Z namespaces/openshift-network-diagnostics/pods/network-check-target-fncdg/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.320959944Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321021946Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321096837Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321109117Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321155818Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321304841Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321369273Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321428454Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321552246Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321617998Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321679489Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3217449Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321810171Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321892623Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.321947984Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322017406Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322069837Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322207859Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322334722Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322451044Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322509916Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322565787Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322633308Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3226906Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322805742Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322866433Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.322922764Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323011686Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323059607Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323068537Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323097178Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323155299Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323302382Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323436235Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323540607Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323620268Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32370367Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323781272Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.323931495Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324039437Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324077528Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324124819Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324163039Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324256111Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324305302Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324336813Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324391424Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324435595Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324515537Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324592218Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324642129Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32468636Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324729211Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324770822Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324815783Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324825993Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.324867064Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.325011417Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.325194691Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.325393364Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32564473Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.325861104Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.325922225Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.325973846Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.325985187Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.325997627Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326069168Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326110009Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32616427Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326237982Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326310543Z namespaces/openshift-network-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326366684Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326426265Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326472046Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326527458Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326593329Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326740322Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326853154Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326936396Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.326982007Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327019777Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327086299Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3271261Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327181251Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327231582Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327325034Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327363995Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327400105Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327483347Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327541688Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327554228Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327584479Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32763876Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.327681761Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.328039558Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32812417Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.328333844Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.328405826Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32859602Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.328677191Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.328802084Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.328865335Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.328906386Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.328996928Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329052909Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.32910565Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329164021Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329243743Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329292224Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329338675Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329383096Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329419826Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329533419Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329640751Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329712143Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329770204Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329837535Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329879066Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329937667Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.329985018Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.330063399Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.330120181Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.330202582Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.330221093Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.330253163Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.330425857Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.330620271Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.330771444Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33103859Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331273784Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331284475Z namespaces/openshift-network-operator/pods/iptables-alerter-2vfdv/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331318705Z namespaces/openshift-network-operator/pods/iptables-alerter-2vfdv/iptables-alerter-2vfdv.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331407367Z namespaces/openshift-network-operator/pods/iptables-alerter-2vfdv/iptables-alerter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331415467Z namespaces/openshift-network-operator/pods/iptables-alerter-2vfdv/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331421277Z namespaces/openshift-network-operator/pods/iptables-alerter-2vfdv/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331460578Z namespaces/openshift-network-operator/pods/iptables-alerter-2vfdv/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33156854Z namespaces/openshift-network-operator/pods/iptables-alerter-2vfdv/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331629462Z namespaces/openshift-network-operator/pods/iptables-alerter-2vfdv/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331662222Z namespaces/openshift-network-operator/pods/iptables-alerter-lw92s/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331704013Z namespaces/openshift-network-operator/pods/iptables-alerter-lw92s/iptables-alerter-lw92s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331776445Z namespaces/openshift-network-operator/pods/iptables-alerter-lw92s/iptables-alerter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331784345Z namespaces/openshift-network-operator/pods/iptables-alerter-lw92s/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331788635Z namespaces/openshift-network-operator/pods/iptables-alerter-lw92s/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331814395Z namespaces/openshift-network-operator/pods/iptables-alerter-lw92s/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331905687Z namespaces/openshift-network-operator/pods/iptables-alerter-lw92s/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.331970318Z namespaces/openshift-network-operator/pods/iptables-alerter-lw92s/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332017439Z namespaces/openshift-network-operator/pods/iptables-alerter-pzwrj/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33204786Z namespaces/openshift-network-operator/pods/iptables-alerter-pzwrj/iptables-alerter-pzwrj.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332125422Z namespaces/openshift-network-operator/pods/iptables-alerter-pzwrj/iptables-alerter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332136762Z namespaces/openshift-network-operator/pods/iptables-alerter-pzwrj/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332147892Z namespaces/openshift-network-operator/pods/iptables-alerter-pzwrj/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332162932Z namespaces/openshift-network-operator/pods/iptables-alerter-pzwrj/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332275575Z namespaces/openshift-network-operator/pods/iptables-alerter-pzwrj/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332346066Z namespaces/openshift-network-operator/pods/iptables-alerter-pzwrj/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332386237Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332428868Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332474779Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332507179Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33255327Z namespaces/openshift-node/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332561921Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33256601Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332608902Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332756384Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.332933848Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.333069691Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.333303196Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.333727544Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.333772305Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.333843507Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.333883977Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.333933348Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33399055Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334110412Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334229334Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334315366Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334385598Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334440519Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33449636Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334526801Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334614262Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334670073Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334734925Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334849857Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334907678Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.334957139Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.335099012Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.335245375Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.335353327Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.335436509Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.335537211Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.335668124Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.335852528Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.335924619Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33596248Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336024721Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336057762Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336104203Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336141373Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336284746Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336370958Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336413349Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33645884Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336508491Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336544822Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336612763Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336649094Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336718455Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336726285Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336762116Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.336908509Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.337100603Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.337259666Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33745519Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.337641914Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.337669405Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.337734106Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.337774557Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.337830848Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.337863189Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33792917Z namespaces/openshift-operators/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33794093Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33794588Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3379587Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.338108923Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.338315758Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33845383Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.338642604Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339051982Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339089813Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339150915Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339217576Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339279517Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339337128Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3393964Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339458411Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339671675Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339747887Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339864719Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.33992117Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.339973331Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340102294Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340141575Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340226277Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340272627Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3403948Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340461821Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340519563Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340618885Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340676926Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340688896Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340735697Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.340822529Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34087431Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.341164586Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.341287168Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.341598765Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.341725447Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342234287Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342318809Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342482203Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342534724Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342580385Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342661786Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342800659Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34283905Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342928072Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.342973962Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343015913Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343069955Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343111205Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343188697Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343207717Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343284769Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343294289Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34332303Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34336891Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343415411Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343509893Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343595315Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343629626Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343693787Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343758469Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34382449Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343939352Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.343996273Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.344065545Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.344077095Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.344103265Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.344229598Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34430244Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.344354861Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.344411962Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.344419782Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.344451063Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34482551Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345018464Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345150127Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345436553Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345618566Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345629887Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345689208Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovnkube-node-74rhp.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345867011Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-node/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345876872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345881892Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.345903752Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346074326Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346140697Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346195768Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346208348Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346212528Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346241549Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346395232Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346468324Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346493404Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/nbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346499174Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/nbdb/nbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346503274Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/nbdb/nbdb/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346545415Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/nbdb/nbdb/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346672968Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346758639Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34678613Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/northd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34679083Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/northd/northd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34679456Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/northd/northd/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346841651Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/northd/northd/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.346945903Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347013145Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/northd/northd/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347053376Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-acl-logging/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347059326Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347063326Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347088856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347193208Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34726632Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347301591Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347310721Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-controller/ovn-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347318591Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347349802Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347683848Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.34776439Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347804391Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovnkube-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347810301Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347847952Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.347891053Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350002366Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350068377Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350105888Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/sbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350115708Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/sbdb/sbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350120838Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/sbdb/sbdb/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350162879Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/sbdb/sbdb/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350290442Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350352023Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-74rhp/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350390733Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350455665Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovnkube-node-gwwvs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350606038Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-node/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350615778Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350621338Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350646459Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350810282Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350878154Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350909974Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350919664Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350923855Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.350953065Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351109718Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.35119683Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.35121907Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/nbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351227871Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/nbdb/nbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351232301Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/nbdb/nbdb/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351278062Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/nbdb/nbdb/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351371064Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351439285Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351478396Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/northd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351485896Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/northd/northd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351499346Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/northd/northd/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351511136Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/northd/northd/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351641109Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.35169941Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/northd/northd/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351743461Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-acl-logging/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351752901Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351757481Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351786432Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351873894Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351942155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351974026Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351981566Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-controller/ovn-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.351985706Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.352007726Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.352442085Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.352513857Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.352535367Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovnkube-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.352541017Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.352545337Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.352586048Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.354762082Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.354855255Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.354899525Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/sbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.354908496Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/sbdb/sbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.354912586Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/sbdb/sbdb/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.354966157Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/sbdb/sbdb/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.35510833Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355226602Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-gwwvs/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355266633Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355337564Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovnkube-node-nld5j.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355508358Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-node/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355519888Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355525178Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355553289Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355731542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355799584Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355841204Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355849305Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355853345Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.355876305Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.35608784Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356156451Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356198722Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/nbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356214842Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/nbdb/nbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356219812Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/nbdb/nbdb/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356261363Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/nbdb/nbdb/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356391076Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356494058Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356511958Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/northd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356516618Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/northd/northd/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356529729Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/northd/northd/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356570819Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/northd/northd/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356703762Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356811624Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/northd/northd/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356841785Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-acl-logging/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356852245Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356857015Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356898226Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.356999158Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.357064489Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.35707439Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-controller/ovn-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.35708088Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.357139491Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.357505409Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.35758489Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.357615371Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovnkube-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.357626381Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.357632141Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.357665742Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.359765804Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.359842356Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.359863557Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/sbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.359874167Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/sbdb/sbdb/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.359881947Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/sbdb/sbdb/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.359925938Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/sbdb/sbdb/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36001708Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360083231Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nld5j/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360097951Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360148982Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360276275Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360290035Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360298355Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360406508Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36053074Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360573581Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360617292Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360699334Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360791975Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360857447Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360914418Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360984769Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.360992749Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36099694Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36104535Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.361126452Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.361134332Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.361214854Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.361460489Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.361748405Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.361886098Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.362244735Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.362456679Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36249261Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.362560592Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.362599812Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.362650803Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.362688724Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.362766846Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.362871968Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36296629Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363003161Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363040181Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363093262Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363129763Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363233255Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363271836Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363312697Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363405529Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363451899Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36346034Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36348469Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363556152Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363586752Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363706115Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.363781096Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36396698Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364043162Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364155044Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364307547Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364484371Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364540812Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364571972Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364633243Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364669594Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364723855Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364767446Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364844488Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36492688Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.364989981Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365054832Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365133174Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365217946Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365326138Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365377019Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36543635Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36544803Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365470391Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365645164Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365834738Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.365986111Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.366231926Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.366456361Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.366466811Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-jht7q/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.366514512Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-jht7q/service-ca-operator-5d6c75f84f-jht7q.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.366620444Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-jht7q/service-ca-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.366629264Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-jht7q/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.366633884Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-jht7q/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.366685515Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-jht7q/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36689937Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-jht7q/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367095104Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367147115Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367237237Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367276637Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367327309Z namespaces/openshift-service-ca/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367375239Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367436231Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367457301Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367537453Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367549753Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367651875Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367749557Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.367846829Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368002522Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368041103Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368087764Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368125735Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368225947Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368276988Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368318229Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36839908Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368441201Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368452361Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368472982Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368527483Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368575524Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368675556Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368751407Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.36885957Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.368937621Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369048853Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369186216Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369422501Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369491402Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369553404Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369617655Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369678506Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369738167Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369805689Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.369922451Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370020933Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370085325Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370155516Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370224507Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370270628Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370317969Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370325939Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37035549Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370549104Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370775359Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.370942052Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371139966Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37134871Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37136106Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-q2vvq/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371384131Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-q2vvq/service-ca-658d7b7c78-q2vvq.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371469893Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-q2vvq/service-ca-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371479663Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-q2vvq/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371484053Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-q2vvq/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371529674Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-q2vvq/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371771119Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-q2vvq/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37183216Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-q2vvq/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371877721Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371897431Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.371955893Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372000293Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372053005Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372098956Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372155207Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372206188Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372263149Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372291699Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372379521Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372457093Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372536815Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372581165Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372625146Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372668787Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372712058Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37278764Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37283239Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372868481Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372950313Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.372996964Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373038015Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373138247Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373233929Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37331566Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373392682Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373470843Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373594916Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373755229Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37380422Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373845481Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373891542Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373934993Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.373977534Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374018165Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374099186Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374197158Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37427203Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374326631Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374396342Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374438983Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374489414Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374497995Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374533225Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374679428Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.374979524Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.375115657Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.375397253Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.375613937Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.375674599Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37573259Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.375779841Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.375827311Z namespaces/openshift/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.375877663Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.375971505Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376023086Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376101597Z namespaces/openshift/apps/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376151058Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376322412Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376433364Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376548216Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376613498Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376671299Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37673688Z namespaces/openshift/batch/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376794961Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376909413Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.376978945Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.377037096Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.377152618Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.37724182Z namespaces/openshift/core/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.377299601Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.377432614Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.377545877Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.377661249Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.377771061Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.377888764Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.378049987Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.378292892Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.378362703Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.378420344Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.378474685Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.378517856Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.379812063Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.379830103Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.379926485Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.380012817Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.380119819Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.380243152Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.380330663Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.380477786Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.38064015Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.380769582Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.380855864Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.380949916Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381069628Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381186261Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381316953Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381397995Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381488287Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381588689Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381704521Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381816553Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.381936306Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.382072909Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.382225832Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.382355955Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.382475867Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.38261072Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.382735242Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.382839454Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.382973707Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.38311962Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.383279483Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.383424716Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.3835909Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.383723322Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.383879146Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.384010178Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.384188972Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.384456457Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.384548269Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.384671672Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.384775054Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.384890986Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385024729Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385120401Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385239583Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385371846Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385509299Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385618701Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385731303Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385826385Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.385910267Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386007419Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386129361Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386277944Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386397957Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386508569Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386628711Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386719573Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386811355Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.386928718Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387088471Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387126232Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387241464Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387328006Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387375297Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387422988Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387466109Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.38751564Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387564501Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387574241Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387615422Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387765175Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.387944518Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388080991Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388360417Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388560221Z namespaces/openshift/policy/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388603642Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388652533Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388706524Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388773695Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388781585Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388831446Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.388953779Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.389104252Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.389256145Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.389369227Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.389595352Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.389719954Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.389840327Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.389949639Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.390078102Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.390189414Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.390333037Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.390456209Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.390599362Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.390707934Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.390824237Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.390920569Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.391060342Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.391186354Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.391326487Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.391435789Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.391547762Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.391653644Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.391758516Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.391858838Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.39195789Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.392064452Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.392272546Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.392388189Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.392498011Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.392599973Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.392708565Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.392836308Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.393003731Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.393155535Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.393294817Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.39340916Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.393509692Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.393615524Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.393730526Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.393832338Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.393953091Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.394062493Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.394187925Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.394337159Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.394464941Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.394618314Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.394746297Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.394868399Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.394990772Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.395089854Z network_logs/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.395124815Z network_logs/cluster_scale [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.395231537Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.395307588Z network_logs/multi-networkpolicy [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.39538787Z network_logs/net-attach-def [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.395473171Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.395632855Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.395723787Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.398092805Z nodes/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.398116276Z nodes/debug [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.398252588Z nodes/ip-10-0-131-114.ec2.internal/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.398290959Z nodes/ip-10-0-131-114.ec2.internal/cpu_affinities.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.398638746Z nodes/ip-10-0-131-114.ec2.internal/dmesg [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.398916952Z nodes/ip-10-0-131-114.ec2.internal/ethtool_channels [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.398996654Z nodes/ip-10-0-131-114.ec2.internal/ethtool_features [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.399089815Z nodes/ip-10-0-131-114.ec2.internal/ip-10-0-131-114.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.399561585Z nodes/ip-10-0-131-114.ec2.internal/irq_affinities.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.399684217Z nodes/ip-10-0-131-114.ec2.internal/lscpu [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.39981913Z nodes/ip-10-0-131-114.ec2.internal/lspci [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.399908092Z nodes/ip-10-0-131-114.ec2.internal/podresources.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.400016914Z nodes/ip-10-0-131-114.ec2.internal/pods_info.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.400137397Z nodes/ip-10-0-131-114.ec2.internal/proc_cmdline [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.400270509Z nodes/ip-10-0-131-114.ec2.internal/sysinfo.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.402062296Z nodes/ip-10-0-131-114.ec2.internal/sysinfo.tgz [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.402220829Z nodes/ip-10-0-131-74.ec2.internal/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.4022416Z nodes/ip-10-0-131-74.ec2.internal/cpu_affinities.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.402528395Z nodes/ip-10-0-131-74.ec2.internal/dmesg [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.4027885Z nodes/ip-10-0-131-74.ec2.internal/ethtool_channels [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.402857502Z nodes/ip-10-0-131-74.ec2.internal/ethtool_features [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.402944924Z nodes/ip-10-0-131-74.ec2.internal/ip-10-0-131-74.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.403395983Z nodes/ip-10-0-131-74.ec2.internal/irq_affinities.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.403564096Z nodes/ip-10-0-131-74.ec2.internal/lscpu [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.403676419Z nodes/ip-10-0-131-74.ec2.internal/lspci [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.403775731Z nodes/ip-10-0-131-74.ec2.internal/podresources.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.403874233Z nodes/ip-10-0-131-74.ec2.internal/pods_info.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.403983565Z nodes/ip-10-0-131-74.ec2.internal/proc_cmdline [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.404064217Z nodes/ip-10-0-131-74.ec2.internal/sysinfo.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.405869803Z nodes/ip-10-0-131-74.ec2.internal/sysinfo.tgz [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.406091538Z nodes/ip-10-0-135-40.ec2.internal/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.406135069Z nodes/ip-10-0-135-40.ec2.internal/cpu_affinities.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.406432605Z nodes/ip-10-0-135-40.ec2.internal/dmesg [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.40668235Z nodes/ip-10-0-135-40.ec2.internal/ethtool_channels [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.406757532Z nodes/ip-10-0-135-40.ec2.internal/ethtool_features [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.406834963Z nodes/ip-10-0-135-40.ec2.internal/ip-10-0-135-40.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.407279552Z nodes/ip-10-0-135-40.ec2.internal/irq_affinities.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.407427005Z nodes/ip-10-0-135-40.ec2.internal/lscpu [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.407563308Z nodes/ip-10-0-135-40.ec2.internal/lspci [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.407646639Z nodes/ip-10-0-135-40.ec2.internal/podresources.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.407753562Z nodes/ip-10-0-135-40.ec2.internal/pods_info.json [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.407863004Z nodes/ip-10-0-135-40.ec2.internal/proc_cmdline [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.407977716Z nodes/ip-10-0-135-40.ec2.internal/sysinfo.log [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.409892435Z nodes/ip-10-0-135-40.ec2.internal/sysinfo.tgz [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.410055999Z pod_network_connectivity_check/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.41010274Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.410262163Z static-pods/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.410275793Z static-pods/kube-apiserver/ [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.419356998Z [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.419377918Z sent 71,848 bytes received 8,569,266 bytes 5,760,742.67 bytes/sec [must-gather] [must-gather-nk8b4] OUT 2026-07-27T19:15:17.419385839Z total size is 108,287,524 speedup is 12.53 [must-gather] [must-gather ] OUT 2026-07-27T19:15:17.719082732Z namespace/openshift-must-gather-r8n2x deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 328368b6-397a-4aea-a1e7-1353e5a2e7f8 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-7kbx4 [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-7kbx4-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-7kbx4 [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 17065781 Jul 27 19:17 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-7kbx4/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 6849b59] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-7kbx4/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 42881d0..6849b59 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-27T19:17:10.857Z","type":3}]