task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [clone-repo] pr/llmisvc-scc-single-node [clone-repo] https://github.com/wseaton/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 38% (1616/4173) Updating files: 39% (1628/4173) Updating files: 39% (1656/4173) Updating files: 39% (1661/4173) Updating files: 40% (1670/4173) Updating files: 40% (1671/4173) Updating files: 40% (1678/4173) Updating files: 40% (1700/4173) Updating files: 41% (1711/4173) Updating files: 42% (1753/4173) Updating files: 43% (1795/4173) Updating files: 44% (1837/4173) Updating files: 45% (1878/4173) Updating files: 46% (1920/4173) Updating files: 47% (1962/4173) Updating files: 48% (2004/4173) Updating files: 49% (2045/4173) Updating files: 50% (2087/4173) Updating files: 51% (2129/4173) Updating files: 52% (2170/4173) Updating files: 53% (2212/4173) Updating files: 54% (2254/4173) Updating files: 55% (2296/4173) Updating files: 56% (2337/4173) Updating files: 57% (2379/4173) Updating files: 58% (2421/4173) Updating files: 58% (2447/4173) Updating files: 59% (2463/4173) Updating files: 60% (2504/4173) Updating files: 61% (2546/4173) Updating files: 62% (2588/4173) Updating files: 63% (2629/4173) Updating files: 64% (2671/4173) Updating files: 65% (2713/4173) Updating files: 66% (2755/4173) Updating files: 67% (2796/4173) Updating files: 68% (2838/4173) Updating files: 69% (2880/4173) Updating files: 70% (2922/4173) Updating files: 71% (2963/4173) Updating files: 72% (3005/4173) Updating files: 73% (3047/4173) Updating files: 74% (3089/4173) Updating files: 75% (3130/4173) Updating files: 76% (3172/4173) Updating files: 77% (3214/4173) Updating files: 78% (3255/4173) Updating files: 79% (3297/4173) Updating files: 80% (3339/4173) Updating files: 81% (3381/4173) Updating files: 82% (3422/4173) Updating files: 83% (3464/4173) Updating files: 84% (3506/4173) Updating files: 85% (3548/4173) Updating files: 86% (3589/4173) Updating files: 87% (3631/4173) Updating files: 88% (3673/4173) Updating files: 88% (3680/4173) Updating files: 89% (3714/4173) Updating files: 90% (3756/4173) Updating files: 91% (3798/4173) Updating files: 92% (3840/4173) Updating files: 92% (3870/4173) Updating files: 93% (3881/4173) Updating files: 94% (3923/4173) Updating files: 95% (3965/4173) Updating files: 96% (4007/4173) Updating files: 97% (4048/4173) Updating files: 98% (4090/4173) Updating files: 99% (4132/4173) Updating files: 100% (4173/4173) Updating files: 100% (4173/4173), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:87faf3b2af7101528d82e427842de59a92695fb38a20d8ec9c374d7ffd4b9c8a [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:87faf3b2af7101528d82e427842de59a92695fb38a20d8ec9c374d7ffd4b9c8a [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:5f7a8815bf612030a07da68b1896a717f793cbd9906cd048a47af7a5dc8611a5 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:5f7a8815bf612030a07da68b1896a717f793cbd9906cd048a47af7a5dc8611a5 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:8d6e2e90be0ae69728cd0c26b71e085b92e2c4a68d58a0f4cb42cde8ff6fc8ac [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:8d6e2e90be0ae69728cd0c26b71e085b92e2c4a68d58a0f4cb42cde8ff6fc8ac [e2e-llm-inference-service] + make e2e-llmisvc-ocp RUNNING_LOCAL=false [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] make run-e2e-ocp E2E_MARKER="llmisvc_core and cluster_cpu and not pvc_storage" E2E_PARALLELISM=2 E2E_PROFILE=llm-d SETUP_E2E=true [e2e-llm-inference-service] make[1]: Entering directory '/workspace/source' [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] SETUP_E2E="true" \ [e2e-llm-inference-service] OPERATOR_TYPE="" \ [e2e-llm-inference-service] KSERVE_NAMESPACE="kserve" \ [e2e-llm-inference-service] RUNNING_LOCAL="false" \ [e2e-llm-inference-service] PYTEST_ARGS="" \ [e2e-llm-inference-service] ./test/scripts/openshift-ci/run-e2e-tests.sh "llmisvc_core and cluster_cpu and not pvc_storage" "2" "llm-d" [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.22.20) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-2157 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] OPT_125M_OCI_MODEL_URI=oci://quay.io/opendatahub/opt-125m-modelcar:latest [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-2157 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-2157 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make[2]: Entering directory '/workspace/source' [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make[2]: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 302 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pygments (1.2MiB) [e2e-llm-inference-service] Downloading setuptools (1.0MiB) [e2e-llm-inference-service] Downloading kubernetes (4.4MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading pillow (6.6MiB) [e2e-llm-inference-service] Downloading pyarrow (47.7MiB) [e2e-llm-inference-service] Downloading uvloop (3.6MiB) [e2e-llm-inference-service] Downloading portforward (4.0MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading pandas (12.2MiB) [e2e-llm-inference-service] Downloading grpcio (6.9MiB) [e2e-llm-inference-service] Downloading botocore (15.1MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.6MiB) [e2e-llm-inference-service] Downloading numpy (16.1MiB) [e2e-llm-inference-service] Downloading black (1.8MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading pygments [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading pillow [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 116 packages in 2.18s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 116 packages in 1.38s [e2e-llm-inference-service] + aiohappyeyeballs==2.7.1 [e2e-llm-inference-service] + aiohttp==3.14.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.5 [e2e-llm-inference-service] + annotated-types==0.8.0 [e2e-llm-inference-service] + anyio==4.15.1 [e2e-llm-inference-service] + asgiref==3.12.1 [e2e-llm-inference-service] + attrs==26.1.0 [e2e-llm-inference-service] + avro==1.12.2 [e2e-llm-inference-service] + black==26.5.1 [e2e-llm-inference-service] + boto3==1.43.93 [e2e-llm-inference-service] + botocore==1.43.93 [e2e-llm-inference-service] + certifi==2026.7.22 [e2e-llm-inference-service] + cffi==2.1.1 [e2e-llm-inference-service] + charset-normalizer==3.5.1 [e2e-llm-inference-service] + click==8.5.0 [e2e-llm-inference-service] + cloudevents==1.13.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.12.0 [e2e-llm-inference-service] + coverage==7.16.1 [e2e-llm-inference-service] + cryptography==50.0.1 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.11 [e2e-llm-inference-service] + execnet==2.1.2 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.8.0 [e2e-llm-inference-service] + googleapis-common-protos==1.75.3 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.84.0 [e2e-llm-inference-service] + grpcio-testing==1.81.1 [e2e-llm-inference-service] + grpcio-tools==1.81.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.8.0 [e2e-llm-inference-service] + httpx==0.28.1 [e2e-llm-inference-service] + httpx-retries==0.6.0 [e2e-llm-inference-service] + idna==3.19 [e2e-llm-inference-service] + iniconfig==2.3.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.1.0 [e2e-llm-inference-service] + kserve==0.21.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==36.0.3 [e2e-llm-inference-service] + markupsafe==3.0.3 [e2e-llm-inference-service] + multidict==6.8.0 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.1.0 [e2e-llm-inference-service] + numpy==2.3.5 [e2e-llm-inference-service] + oauthlib==3.3.1 [e2e-llm-inference-service] + opentelemetry-api==1.44.0 [e2e-llm-inference-service] + opentelemetry-exporter-otlp==1.44.0 [e2e-llm-inference-service] + opentelemetry-exporter-otlp-proto-common==1.44.0 [e2e-llm-inference-service] + opentelemetry-exporter-otlp-proto-grpc==1.44.0 [e2e-llm-inference-service] + opentelemetry-exporter-otlp-proto-http==1.44.0 [e2e-llm-inference-service] + opentelemetry-instrumentation==0.65b0 [e2e-llm-inference-service] + opentelemetry-instrumentation-asgi==0.65b0 [e2e-llm-inference-service] + opentelemetry-instrumentation-fastapi==0.65b0 [e2e-llm-inference-service] + opentelemetry-instrumentation-grpc==0.65b0 [e2e-llm-inference-service] + opentelemetry-instrumentation-logging==0.65b0 [e2e-llm-inference-service] + opentelemetry-proto==1.44.0 [e2e-llm-inference-service] + opentelemetry-sdk==1.44.0 [e2e-llm-inference-service] + opentelemetry-semantic-conventions==0.65b0 [e2e-llm-inference-service] + opentelemetry-util-http==0.65b0 [e2e-llm-inference-service] + orjson==3.12.0 [e2e-llm-inference-service] + packaging==26.3 [e2e-llm-inference-service] + pandas==2.3.3 [e2e-llm-inference-service] + pathspec==1.1.1 [e2e-llm-inference-service] + pillow==12.3.0 [e2e-llm-inference-service] + platformdirs==4.11.8 [e2e-llm-inference-service] + pluggy==1.6.0 [e2e-llm-inference-service] + portforward==0.7.6 [e2e-llm-inference-service] + prometheus-client==0.26.0 [e2e-llm-inference-service] + propcache==0.5.2 [e2e-llm-inference-service] + protobuf==6.33.6 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==25.0.1 [e2e-llm-inference-service] + pyasn1==0.6.4 [e2e-llm-inference-service] + pycparser==3.0 [e2e-llm-inference-service] + pydantic==2.13.5 [e2e-llm-inference-service] + pydantic-core==2.46.5 [e2e-llm-inference-service] + pygments==2.21.0 [e2e-llm-inference-service] + pyjwt==2.14.0 [e2e-llm-inference-service] + pytest==9.1.1 [e2e-llm-inference-service] + pytest-asyncio==1.4.0 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.36.2 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.8.0 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.2.3 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytokens==0.4.1 [e2e-llm-inference-service] + pytz==2026.3.post1 [e2e-llm-inference-service] + pyyaml==6.0.3 [e2e-llm-inference-service] + requests==2.34.2 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + s3transfer==0.19.2 [e2e-llm-inference-service] + setuptools==80.10.2 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + starlette==1.6.0 [e2e-llm-inference-service] + tabulate==0.10.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.2 [e2e-llm-inference-service] + tomlkit==0.15.1 [e2e-llm-inference-service] + typing-extensions==4.16.0 [e2e-llm-inference-service] + typing-inspection==0.4.4 [e2e-llm-inference-service] + tzdata==2026.4 [e2e-llm-inference-service] + urllib3==2.8.0 [e2e-llm-inference-service] + uvicorn==0.53.0 [e2e-llm-inference-service] + uvloop==0.22.1 [e2e-llm-inference-service] + watchfiles==1.2.0 [e2e-llm-inference-service] + websocket-client==1.9.2 [e2e-llm-inference-service] + websockets==17.1 [e2e-llm-inference-service] + wrapt==2.4.1 [e2e-llm-inference-service] + yarl==1.24.5 [e2e-llm-inference-service] Audited 1 package in 48ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make[2]: Entering directory '/workspace/source' [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] make[2]: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-4 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-4 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-4 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-4 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-4 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-4 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-7dc85965f-g2fhk 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-7dc85965f-g2fhk condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-86bd856cfb-8k48v 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-86bd856cfb-8k48v condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-7b7c985c58-r69ft 1/1 Running 0 51s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-7b7c985c58-r69ft condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.22.20) [e2e-llm-inference-service] 🎯 Server version (4.22.20) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.1 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.1 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-8465994787-n5xg6 1/1 Running 0 24s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-8465994787-n5xg6 condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-7876bfcb74-ldwtr 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-7876bfcb74-ldwtr condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-86b669b864-4wfkx 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-86b669b864-4wfkx condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-66498fdf6c-b8722 1/1 Running 0 94s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-66498fdf6c-b8722 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] RHCL kuadrant-system/rhcl-operator: installed version 1.4.3; audit logging minimum 1.5.1 [e2e-llm-inference-service] ℹ️ skipping LLMInferenceService audit logging setup: installed RHCL 1.4.3 is below the minimum 1.5.1 (release the Kuadrant audit logging fixes are planned for); the installed RHCL build has no TelemetryPolicy logging support (missing: Authorino.spec.enableLoggingFields, TelemetryPolicy.spec.logging) [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-66498fdf6c-b8722 1/1 Running 0 104s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-66498fdf6c-b8722 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for the authorino Deployment to reflect the applied spec… [e2e-llm-inference-service] ⏳ waiting for the authorino operand to roll out… [e2e-llm-inference-service] deployment "authorino" successfully rolled out [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] error: open /credentials/kubeconfig: read-only file system [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make[2]: Entering directory '/workspace/source' [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] make[2]: Leaving directory '/workspace/source' [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:5f7a8815bf612030a07da68b1896a717f793cbd9906cd048a47af7a5dc8611a5 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:8d6e2e90be0ae69728cd0c26b71e085b92e2c4a68d58a0f4cb42cde8ff6fc8ac [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:87faf3b2af7101528d82e427842de59a92695fb38a20d8ec9c374d7ffd4b9c8a [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:5f7a8815bf612030a07da68b1896a717f793cbd9906cd048a47af7a5dc8611a5 [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:8d6e2e90be0ae69728cd0c26b71e085b92e2c4a68d58a0f4cb42cde8ff6fc8ac [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:87faf3b2af7101528d82e427842de59a92695fb38a20d8ec9c374d7ffd4b9c8a [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:8bcd69b1e985441df1221f21e42e9da4815f90c40a0b0f824dd1e14e4e393fe7 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] kserve-osc-storage-helper=registry.redhat.io/openshift-sandboxed-containers/osc-storage-helper:1.13.1 [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaii-fast/vllm-cuda-rhel9@sha256:97c7d6f66f3f0897b3ff10ddd135de930d45bdd116cf41a43e3ad7270029116a [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:odh-v3.6.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:odh-v3.6.0 [e2e-llm-inference-service] kserve-llm-d-tokenizer=registry.redhat.io/rhaii/vllm-cpu-rhel9@sha256:0ec0f5ff9a5ff263f58db0cfb19e3d7eef24b94c2d8629fe5145f5a8747e28c9 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-cpu=registry.redhat.io/rhaii/vllm-cpu-rhel9@sha256:0ec0f5ff9a5ff263f58db0cfb19e3d7eef24b94c2d8629fe5145f5a8747e28c9 [e2e-llm-inference-service] kserve-llm-d-cpu-fast-1=registry.redhat.io/rhaii/vllm-cpu-rhel9@sha256:0ec0f5ff9a5ff263f58db0cfb19e3d7eef24b94c2d8629fe5145f5a8747e28c9 [e2e-llm-inference-service] kserve-llm-d-cpu-fast-2=registry.redhat.io/rhaii/vllm-cpu-rhel9@sha256:0ec0f5ff9a5ff263f58db0cfb19e3d7eef24b94c2d8629fe5145f5a8747e28c9 [e2e-llm-inference-service] kserve-llm-d-cpu-upstream-version=0.18.0+rhaiv.11 [e2e-llm-inference-service] kserve-llm-d-cpu-fast-1-upstream-version=0.18.0+rhaiv.11 [e2e-llm-inference-service] kserve-llm-d-cpu-fast-2-upstream-version=0.18.0+rhaiv.11 [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-rbac-proxy@sha256:acd9de62b8339dbacd39635317e5e596e09a70ae26ac9eccdde4a4124388740f [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferencegraph-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llmisvc-confidential-bare-metal serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferencegraph-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-confidential-bare-metal-resources serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llmisvc-confidential-bare-metal serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-confidential-bare-metal serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel-multi-port serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel-multi-port serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-default-pd serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-modality serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-cpu serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-cpu-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-cpu-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-86c86ffb6b-w7p6x 0/1 Running 0 7s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-86c86ffb6b-w7p6x condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferencegraph-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llmisvc-confidential-bare-metal serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferencegraph-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-confidential-bare-metal-resources serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llmisvc-confidential-bare-metal serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-confidential-bare-metal serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel-multi-port serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel-multi-port serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-default-pd serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-modality serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-cpu serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-cpu-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-cpu-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-76d8b77fbb-9rjtf" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-76d8b77fbb-9nwtn condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-openshift-distro-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-openshift-distro-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] networkpolicy.networking.k8s.io/odh-model-controller-webhook-allow-ingress created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-mlserver created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-mlserver-cuda created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-ovms created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-x86 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-x86-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-x86-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cuda created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cuda-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cuda-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-gaudi created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-gaudi-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-gaudi-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-omni-cuda created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-omni-cuda-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-omni-cuda-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-rocm created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-rocm-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-rocm-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-ppc64le created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-ppc64le-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-ppc64le-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-s390x created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-s390x-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-s390x-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-x86 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-x86-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-x86-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-omni-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-omni-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-omni-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-llm-omni created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-llm-sglang created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.22.20 [e2e-llm-inference-service] Kubernetes Version: v1.35.9 [e2e-llm-inference-service] ClusterVersion desired: 4.22.20 [e2e-llm-inference-service] ClusterVersion history (latest): 4.22.20 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.1 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-4 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.3 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.2 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.2 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.3 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] === LLMInferenceService audit logging diagnostics === [e2e-llm-inference-service] RHCL subscription kuadrant-system/rhcl-operator: installedCSV=rhcl-operator.v1.4.3 version=1.4.3 [e2e-llm-inference-service] LLMInferenceService audit logging: mode=auto minimum RHCL version=1.5.1 [e2e-llm-inference-service] Deployment kuadrant-system/kuadrant-operator-controller-manager: [e2e-llm-inference-service] kuadrant-operator-controller-manager: ready=1 image=registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e [e2e-llm-inference-service] imageID: registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:22341c0a543abdcd6cecb58e0e84ffeb713799db22bdf3c01e38cf5c248aee0e [e2e-llm-inference-service] Deployment kuadrant-system/authorino-operator: [e2e-llm-inference-service] authorino-operator: ready=1 image=registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589 [e2e-llm-inference-service] imageID: registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:65471fc61b3422f7ba6579dee62a848442a800d60373d0ae21282d683225e0ff [e2e-llm-inference-service] Deployment kuadrant-system/authorino: [e2e-llm-inference-service] authorino: ready=1 image=registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0 [e2e-llm-inference-service] imageID: registry.redhat.io/rhcl-1/authorino-rhel9@sha256:c4a5a8f911a5097293de58dde320e9b8e51783b8fd92121de73bc60f6672cdf0 [e2e-llm-inference-service] Authorino CRD supports spec.enableLoggingFields: no [e2e-llm-inference-service] Authorino CR (authorino/kuadrant-system) spec: logLevel=debug logMode= enableLoggingFields= loggingFieldsMaxValueBytes= [e2e-llm-inference-service] TelemetryPolicy CRD supports spec.logging: no [e2e-llm-inference-service] TelemetryPolicies (all namespaces): [e2e-llm-inference-service] === End LLMInferenceService audit logging diagnostics === [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:5f7a8815bf612030a07da68b1896a717f793cbd9906cd048a47af7a5dc8611a5 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:5f7a8815bf612030a07da68b1896a717f793cbd9906cd048a47af7a5dc8611a5 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:3cc415a74b16c43c474956cc87c78003407f641260fabce7525afd11e6fdaf24 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:8d6e2e90be0ae69728cd0c26b71e085b92e2c4a68d58a0f4cb42cde8ff6fc8ac [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:8d6e2e90be0ae69728cd0c26b71e085b92e2c4a68d58a0f4cb42cde8ff6fc8ac [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llmisvc_core and cluster_cpu and not pvc_storage [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-9.1.1, pluggy-1.6.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.142.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '9.1.1', 'pluggy': '1.6.0'}, 'Plugins': {'cov': '5.0.0', 'asyncio': '1.4.0', 'xdist': '3.8.0', 'anyio': '4.15.1', 'httpx': '0.36.2', 'metadata': '3.1.1', 'json-report': '1.5.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: cov-5.0.0, asyncio-1.4.0, xdist-3.8.0, anyio-4.15.1, httpx-0.36.2, metadata-3.1.1, json-report-1.5.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [50 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_accelerator_config.py::test_cpu_accelerator_preset_applies_to_workload [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] 2026-10-30 22:31:21.424 7367 kserve INFO [conftest.py:configure_logger():51] trace_id=none Logger configured [e2e-llm-inference-service] 2026-10-30 22:31:21.424 7364 kserve INFO [conftest.py:configure_logger():51] trace_id=none Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_accelerator_config.py::test_cpu_accelerator_preset_applies_to_workload [e2e-llm-inference-service] llmisvc/test_accelerator_config.py::test_cpu_accelerator_preset_user_image_wins [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_accelerator_config.py::test_cpu_accelerator_preset_user_image_wins [e2e-llm-inference-service] llmisvc/test_accelerator_config.py::test_cpu_accelerator_preset_serves_inference[cluster_cpu-cluster_single_node-router-managed-model-fb-opt-125m-workload-non-root] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_accelerator_config.py::test_cpu_accelerator_preset_serves_inference[cluster_cpu-cluster_single_node-router-managed-model-fb-opt-125m-workload-non-root] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-10-30 22:40:38.903 7364 kserve.trace trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-10-30 22:40:38.903 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-10-30 22:40:38.974 7364 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:40:38.974 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:40:38.997 7364 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:40:38.997 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] 2026-10-30 22:42:59.682 7364 kserve.trace trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-10-30 22:42:59.682 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-10-30 22:42:59.759 7364 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:42:59.759 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:42:59.780 7364 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:42:59.780 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_audit_logging.py::test_llm_audit_records_for_authenticated_requests[cluster_cpu-cluster_single_node-audit-auth-enabled] [e2e-llm-inference-service] [gw0] SKIPPED llmisvc/test_llm_audit_logging.py::test_llm_audit_records_for_authenticated_requests[cluster_cpu-cluster_single_node-audit-auth-enabled] [e2e-llm-inference-service] llmisvc/test_llm_audit_logging.py::test_llm_audit_records_anonymous_when_auth_disabled[cluster_cpu-cluster_single_node-audit-auth-disabled] [e2e-llm-inference-service] [gw0] SKIPPED llmisvc/test_llm_audit_logging.py::test_llm_audit_records_anonymous_when_auth_disabled[cluster_cpu-cluster_single_node-audit-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_audit_logging.py::test_llm_audit_security_controls[cluster_cpu-cluster_single_node-audit-security-controls] [e2e-llm-inference-service] [gw0] SKIPPED llmisvc/test_llm_audit_logging.py::test_llm_audit_security_controls[cluster_cpu-cluster_single_node-audit-security-controls] [e2e-llm-inference-service] llmisvc/test_llm_audit_logging.py::test_llm_audit_sensitive_header_field_is_rejected_or_redacted[cluster_cpu-cluster_single_node-audit-redaction] [e2e-llm-inference-service] [gw0] SKIPPED llmisvc/test_llm_audit_logging.py::test_llm_audit_sensitive_header_field_is_rejected_or_redacted[cluster_cpu-cluster_single_node-audit-redaction] [e2e-llm-inference-service] llmisvc/test_llm_audit_logging.py::test_llm_audit_logging_config_is_stable [e2e-llm-inference-service] [gw0] SKIPPED llmisvc/test_llm_audit_logging.py::test_llm_audit_logging_config_is_stable [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-template-args-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-template-args-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-10-30 22:56:45.859 7364 kserve.trace trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-6f0aa991 [e2e-llm-inference-service] 2026-10-30 22:56:45.859 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-6f0aa991 [e2e-llm-inference-service] 2026-10-30 22:56:45.928 7364 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:56:45.928 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:56:45.946 7364 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 22:56:45.946 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-oci] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-oci] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-10-30 23:11:23.816 7364 kserve.trace trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-10-30 23:11:23.816 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-10-30 23:11:23.879 7364 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 23:11:23.879 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 23:11:23.897 7364 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 23:11:23.897 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-10-30 23:11:23.898 7364 kserve.trace trace_id=none Checking HttpRoute router-route-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-10-30 23:11:23.898 7364 kserve.trace INFO [gw_api.py:create_or_update_route():121] trace_id=none Checking HttpRoute router-route-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-10-30 23:11:23.912 7364 kserve.trace trace_id=none Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-10-30 23:11:23.912 7364 kserve.trace INFO [gw_api.py:create_or_update_route():149] trace_id=none Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-10-30 23:11:23.938 7364 kserve.trace trace_id=none ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-10-30 23:11:23.938 7364 kserve.trace INFO [gw_api.py:create_or_update_route():157] trace_id=none ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-10-30 23:11:23.938 7364 kserve.trace trace_id=none Checking HttpRoute router-route-2 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-10-30 23:11:23.938 7364 kserve.trace INFO [gw_api.py:create_or_update_route():121] trace_id=none Checking HttpRoute router-route-2 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-10-30 23:11:23.961 7364 kserve.trace trace_id=none Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-10-30 23:11:23.961 7364 kserve.trace INFO [gw_api.py:create_or_update_route():149] trace_id=none Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-10-30 23:11:23.985 7364 kserve.trace trace_id=none ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-10-30 23:11:23.985 7364 kserve.trace INFO [gw_api.py:create_or_update_route():157] trace_id=none ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-10-30 23:20:47.091 7364 kserve.trace trace_id=none Checking Gateway router-gateway-2 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-10-30 23:20:47.091 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-2 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-pvc] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters-one-pvc] 2026-10-30 23:22:55.593 7364 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-10-30 23:22:55.593 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-10-30 23:22:55.613 7364 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-10-30 23:22:55.613 7364 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-10-30 23:22:55.614 7364 kserve.trace trace_id=none Checking HttpRoute router-route-3 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-10-30 23:22:55.614 7364 kserve.trace INFO [gw_api.py:create_or_update_route():121] trace_id=none Checking HttpRoute router-route-3 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-10-30 23:22:55.629 7364 kserve.trace trace_id=none Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-10-30 23:22:55.629 7364 kserve.trace INFO [gw_api.py:create_or_update_route():149] trace_id=none Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-10-30 23:22:55.652 7364 kserve.trace trace_id=none ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-10-30 23:22:55.652 7364 kserve.trace INFO [gw_api.py:create_or_update_route():157] trace_id=none ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-10-30 23:22:55.652 7364 kserve.trace trace_id=none Checking HttpRoute router-route-4 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-10-30 23:22:55.652 7364 kserve.trace INFO [gw_api.py:create_or_update_route():121] trace_id=none Checking HttpRoute router-route-4 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-10-30 23:22:55.667 7364 kserve.trace trace_id=none Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-10-30 23:22:55.667 7364 kserve.trace INFO [gw_api.py:create_or_update_route():149] trace_id=none Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-10-30 23:22:55.685 7364 kserve.trace trace_id=none ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-10-30 23:22:55.685 7364 kserve.trace INFO [gw_api.py:create_or_update_route():157] trace_id=none ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters-one-pvc] [e2e-llm-inference-service] llmisvc/test_llm_model_routing_only.py::test_model_based_routing_only[cluster_single_node-cluster_cpu-gateway] 2026-10-30 23:25:59.560 7367 kserve.trace trace_id=none Checking Gateway model-routing-only-gateway in namespace e2e-test-model-based-routing-only-f772d59b [e2e-llm-inference-service] 2026-10-30 23:25:59.560 7367 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway model-routing-only-gateway in namespace e2e-test-model-based-routing-only-f772d59b [e2e-llm-inference-service] 2026-10-30 23:25:59.631 7367 kserve.trace trace_id=none Resource not found, creating Gateway model-routing-only-gateway [e2e-llm-inference-service] 2026-10-30 23:25:59.631 7367 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway model-routing-only-gateway [e2e-llm-inference-service] 2026-10-30 23:25:59.650 7367 kserve.trace trace_id=none ✓ Successfully created Gateway model-routing-only-gateway [e2e-llm-inference-service] 2026-10-30 23:25:59.650 7367 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway model-routing-only-gateway [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_model_routing_only.py::test_model_based_routing_only[cluster_single_node-cluster_cpu-gateway] [e2e-llm-inference-service] llmisvc/test_llm_model_routing_only.py::test_model_based_routing_only[cluster_single_node-cluster_cpu-service] 2026-10-30 23:27:16.464 7367 kserve.trace trace_id=none Checking Gateway model-routing-only-gateway in namespace e2e-test-model-based-routing-only-fae765e6 [e2e-llm-inference-service] 2026-10-30 23:27:16.464 7367 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway model-routing-only-gateway in namespace e2e-test-model-based-routing-only-fae765e6 [e2e-llm-inference-service] 2026-10-30 23:27:16.521 7367 kserve.trace trace_id=none Resource not found, creating Gateway model-routing-only-gateway [e2e-llm-inference-service] 2026-10-30 23:27:16.521 7367 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway model-routing-only-gateway [e2e-llm-inference-service] 2026-10-30 23:27:16.537 7367 kserve.trace trace_id=none ✓ Successfully created Gateway model-routing-only-gateway [e2e-llm-inference-service] 2026-10-30 23:27:16.537 7367 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway model-routing-only-gateway [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_model_routing_only.py::test_model_based_routing_only[cluster_single_node-cluster_cpu-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] ___ test_llm_tls_resources[router-managed-workload-pd-cpu-model-fb-opt-125m] ___ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] configuration = [e2e-llm-inference-service] _method = 'GET' [e2e-llm-inference-service] url = 'wss://ae09c812c6e404d9a80b724db9fbfb35-422a5c72c4122098.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/e2e-test-l...ommand=tr+%27%5C0%27+%27%5Cn%27+%3C+%2Fproc%2F1%2Fcmdline&container=main&stderr=True&stdin=False&stdout=True&tty=False' [e2e-llm-inference-service] kwargs = {'binary': False, 'query_params': [('command', ['/bin/bash', '-c', "tr '\\0' '\\n' < /proc/1/cmdline"]), ('container',...('tty', False)], 'headers': {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/36.0.3/python'}, 'post_params': [], ...} [e2e-llm-inference-service] [e2e-llm-inference-service] def websocket_call(configuration, _method, url, **kwargs): [e2e-llm-inference-service] """An internal function to be called in api-client when a websocket [e2e-llm-inference-service] connection is required. method, url, and kwargs are the parameters of [e2e-llm-inference-service] apiClient.request method.""" [e2e-llm-inference-service] [e2e-llm-inference-service] url = get_websocket_url(url, kwargs.get("query_params")) [e2e-llm-inference-service] headers = kwargs.get("headers") [e2e-llm-inference-service] _request_timeout = kwargs.get("_request_timeout", 60) [e2e-llm-inference-service] _preload_content = kwargs.get("_preload_content", True) [e2e-llm-inference-service] capture_all = kwargs.get("capture_all", True) [e2e-llm-inference-service] binary = kwargs.get('binary', False) [e2e-llm-inference-service] try: [e2e-llm-inference-service] > client = WSClient(configuration, url, headers, capture_all, binary=binary) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/stream/ws_client.py:587: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] configuration = [e2e-llm-inference-service] url = 'wss://ae09c812c6e404d9a80b724db9fbfb35-422a5c72c4122098.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/e2e-test-l...ommand=tr+%27%5C0%27+%27%5Cn%27+%3C+%2Fproc%2F1%2Fcmdline&container=main&stderr=True&stdin=False&stdout=True&tty=False' [e2e-llm-inference-service] headers = {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/36.0.3/python'} [e2e-llm-inference-service] capture_all = True, binary = False [e2e-llm-inference-service] [e2e-llm-inference-service] def __init__(self, configuration, url, headers, capture_all, binary=False): [e2e-llm-inference-service] """A websocket client with support for channels. [e2e-llm-inference-service] [e2e-llm-inference-service] Exec command uses different channels for different streams. for [e2e-llm-inference-service] example, 0 is stdin, 1 is stdout and 2 is stderr. Some other API calls [e2e-llm-inference-service] like port forwarding can forward different pods' streams to different [e2e-llm-inference-service] channels. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self._connected = False [e2e-llm-inference-service] self._channels = {} [e2e-llm-inference-service] self._closed_channels = set() [e2e-llm-inference-service] self.subprotocol = None [e2e-llm-inference-service] self.binary = binary [e2e-llm-inference-service] self.newline = '\n' if not self.binary else b'\n' [e2e-llm-inference-service] if capture_all: [e2e-llm-inference-service] self._all = StringIO() if not self.binary else BytesIO() [e2e-llm-inference-service] else: [e2e-llm-inference-service] self._all = _IgnoredIO() [e2e-llm-inference-service] > self.sock = create_websocket(configuration, url, headers) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/stream/ws_client.py:70: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] configuration = [e2e-llm-inference-service] url = 'wss://ae09c812c6e404d9a80b724db9fbfb35-422a5c72c4122098.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/e2e-test-l...ommand=tr+%27%5C0%27+%27%5Cn%27+%3C+%2Fproc%2F1%2Fcmdline&container=main&stderr=True&stdin=False&stdout=True&tty=False' [e2e-llm-inference-service] headers = {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/36.0.3/python'} [e2e-llm-inference-service] [e2e-llm-inference-service] def create_websocket(configuration, url, headers=None): [e2e-llm-inference-service] enableTrace(False) [e2e-llm-inference-service] [e2e-llm-inference-service] # We just need to pass the Authorization, ignore all the other [e2e-llm-inference-service] # http headers we get from the generated code [e2e-llm-inference-service] header = [] [e2e-llm-inference-service] if headers and 'authorization' in headers: [e2e-llm-inference-service] header.append("authorization: %s" % headers['authorization']) [e2e-llm-inference-service] if headers and 'sec-websocket-protocol' in headers: [e2e-llm-inference-service] header.append("sec-websocket-protocol: %s" % [e2e-llm-inference-service] headers['sec-websocket-protocol']) [e2e-llm-inference-service] else: [e2e-llm-inference-service] header.append("sec-websocket-protocol: %s,%s" % (V5_CHANNEL_PROTOCOL, V4_CHANNEL_PROTOCOL)) [e2e-llm-inference-service] [e2e-llm-inference-service] if url.startswith('wss://') and configuration.verify_ssl: [e2e-llm-inference-service] ssl_opts = { [e2e-llm-inference-service] 'cert_reqs': ssl.CERT_REQUIRED, [e2e-llm-inference-service] 'ca_certs': configuration.ssl_ca_cert or certifi.where(), [e2e-llm-inference-service] } [e2e-llm-inference-service] if configuration.assert_hostname is not None: [e2e-llm-inference-service] ssl_opts['check_hostname'] = configuration.assert_hostname [e2e-llm-inference-service] else: [e2e-llm-inference-service] ssl_opts = {'cert_reqs': ssl.CERT_NONE} [e2e-llm-inference-service] [e2e-llm-inference-service] if configuration.cert_file: [e2e-llm-inference-service] ssl_opts['certfile'] = configuration.cert_file [e2e-llm-inference-service] if configuration.key_file: [e2e-llm-inference-service] ssl_opts['keyfile'] = configuration.key_file [e2e-llm-inference-service] if configuration.tls_server_name: [e2e-llm-inference-service] ssl_opts['server_hostname'] = configuration.tls_server_name [e2e-llm-inference-service] [e2e-llm-inference-service] websocket = WebSocket(sslopt=ssl_opts, skip_utf8_validation=False) [e2e-llm-inference-service] connect_opt = { [e2e-llm-inference-service] 'header': header [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] if configuration.proxy or configuration.proxy_headers: [e2e-llm-inference-service] connect_opt = websocket_proxycare(connect_opt, configuration, url, headers) [e2e-llm-inference-service] [e2e-llm-inference-service] > websocket.connect(url, **connect_opt) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/stream/ws_client.py:553: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'wss://ae09c812c6e404d9a80b724db9fbfb35-422a5c72c4122098.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/e2e-test-l...ommand=tr+%27%5C0%27+%27%5Cn%27+%3C+%2Fproc%2F1%2Fcmdline&container=main&stderr=True&stdin=False&stdout=True&tty=False' [e2e-llm-inference-service] options = {'header': ['sec-websocket-protocol: v5.channel.k8s.io,v4.channel.k8s.io']} [e2e-llm-inference-service] [e2e-llm-inference-service] def connect(self, url, **options): [e2e-llm-inference-service] """ [e2e-llm-inference-service] Connect to url. url is websocket url scheme. [e2e-llm-inference-service] ie. ws://host:port/resource [e2e-llm-inference-service] You can customize using 'options'. [e2e-llm-inference-service] If you set "header" list object, you can set your own custom header. [e2e-llm-inference-service] [e2e-llm-inference-service] >>> ws = WebSocket() [e2e-llm-inference-service] >>> ws.connect("ws://websockets.chilkat.io/wsChilkatEcho.ashx", [e2e-llm-inference-service] ... header=["User-Agent: MyProgram", [e2e-llm-inference-service] ... "x-custom: header"]) [e2e-llm-inference-service] [e2e-llm-inference-service] Parameters [e2e-llm-inference-service] ---------- [e2e-llm-inference-service] header: list or dict [e2e-llm-inference-service] Custom http header list or dict. [e2e-llm-inference-service] cookie: str [e2e-llm-inference-service] Cookie value. [e2e-llm-inference-service] origin: str [e2e-llm-inference-service] Custom origin url. [e2e-llm-inference-service] connection: str [e2e-llm-inference-service] Custom connection header value. [e2e-llm-inference-service] Default value "Upgrade" set in _handshake.py [e2e-llm-inference-service] suppress_origin: bool [e2e-llm-inference-service] Suppress outputting origin header. [e2e-llm-inference-service] suppress_host: bool [e2e-llm-inference-service] Suppress outputting host header. [e2e-llm-inference-service] host: str [e2e-llm-inference-service] Custom host header string. [e2e-llm-inference-service] timeout: int or float [e2e-llm-inference-service] Socket timeout time. This value is an integer or float. [e2e-llm-inference-service] If you set None for this value, it means "use default_timeout value" [e2e-llm-inference-service] http_proxy_host: str [e2e-llm-inference-service] HTTP proxy host name. [e2e-llm-inference-service] http_proxy_port: str or int [e2e-llm-inference-service] HTTP proxy port. Required when http_proxy_host is set. Proxies [e2e-llm-inference-service] from environment variables default to port 80. [e2e-llm-inference-service] http_no_proxy: list [e2e-llm-inference-service] Whitelisted host names that don't use the proxy. [e2e-llm-inference-service] http_proxy_auth: tuple [e2e-llm-inference-service] HTTP proxy auth information. Tuple of username and password. Default is None. [e2e-llm-inference-service] http_proxy_timeout: int or float [e2e-llm-inference-service] HTTP proxy timeout, default is 60 sec as per python-socks. [e2e-llm-inference-service] redirect_limit: int [e2e-llm-inference-service] Number of redirects to follow. [e2e-llm-inference-service] subprotocols: list [e2e-llm-inference-service] List of available subprotocols. Default is None. [e2e-llm-inference-service] socket: socket [e2e-llm-inference-service] Pre-initialized stream socket. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self.sock_opt.timeout = options.get("timeout", self.sock_opt.timeout) [e2e-llm-inference-service] > self.sock, addrs = connect( [e2e-llm-inference-service] url, self.sock_opt, proxy_info(**options), options.pop("socket", None) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/websocket/_core.py:279: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] url = 'wss://ae09c812c6e404d9a80b724db9fbfb35-422a5c72c4122098.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/e2e-test-l...ommand=tr+%27%5C0%27+%27%5Cn%27+%3C+%2Fproc%2F1%2Fcmdline&container=main&stderr=True&stdin=False&stdout=True&tty=False' [e2e-llm-inference-service] options = [e2e-llm-inference-service] proxy = , socket = None [e2e-llm-inference-service] [e2e-llm-inference-service] def connect( [e2e-llm-inference-service] url: str, options: Any, proxy: Any, socket: Any [e2e-llm-inference-service] ) -> Tuple[socket.socket, Tuple[str, int, str]]: [e2e-llm-inference-service] # Use _start_proxied_socket() only for socks4 or socks5 proxy [e2e-llm-inference-service] # Use _tunnel() for http proxy [e2e-llm-inference-service] # TODO: Use python-socks for http protocol also, to standardize flow [e2e-llm-inference-service] if proxy.proxy_host and not socket and proxy.proxy_protocol != "http": [e2e-llm-inference-service] return _start_proxied_socket(url, options, proxy) [e2e-llm-inference-service] [e2e-llm-inference-service] hostname, port_from_url, resource, is_secure = parse_url(url) [e2e-llm-inference-service] [e2e-llm-inference-service] if socket: [e2e-llm-inference-service] return socket, (hostname, port_from_url, resource) [e2e-llm-inference-service] [e2e-llm-inference-service] addrinfo_list, need_tunnel, auth = _get_addrinfo_list( [e2e-llm-inference-service] hostname, port_from_url, is_secure, proxy [e2e-llm-inference-service] ) [e2e-llm-inference-service] if not addrinfo_list: [e2e-llm-inference-service] raise WebSocketException(f"Host not found.: {hostname}:{port_from_url}") [e2e-llm-inference-service] [e2e-llm-inference-service] sock = None [e2e-llm-inference-service] try: [e2e-llm-inference-service] > sock = _open_socket(addrinfo_list, options.sockopt, options.timeout) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/websocket/_http.py:158: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] addrinfo_list = [(, , 6, '', ('3.22.17.175', 6443)), (,... '', ('3.148.74.118', 6443)), (, , 6, '', ('16.58.179.28', 6443))] [e2e-llm-inference-service] sockopt = [], timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def _open_socket(addrinfo_list, sockopt, timeout): [e2e-llm-inference-service] err = None [e2e-llm-inference-service] for addrinfo in addrinfo_list: [e2e-llm-inference-service] family, socktype, proto = addrinfo[:3] [e2e-llm-inference-service] sock = socket.socket(family, socktype, proto) [e2e-llm-inference-service] sock.settimeout(timeout) [e2e-llm-inference-service] for opts in DEFAULT_SOCKET_OPTION: [e2e-llm-inference-service] sock.setsockopt(*opts) [e2e-llm-inference-service] for opts in sockopt: [e2e-llm-inference-service] sock.setsockopt(*opts) [e2e-llm-inference-service] [e2e-llm-inference-service] address = addrinfo[4] [e2e-llm-inference-service] err = None [e2e-llm-inference-service] while not err: [e2e-llm-inference-service] try: [e2e-llm-inference-service] sock.connect(address) [e2e-llm-inference-service] except socket.error as error: [e2e-llm-inference-service] sock.close() [e2e-llm-inference-service] error.remote_ip = str(address[0]) # type: ignore[attr-defined] [e2e-llm-inference-service] eConnRefused = ( [e2e-llm-inference-service] errno.ECONNREFUSED, [e2e-llm-inference-service] getattr(errno, "WSAECONNREFUSED", errno.ECONNREFUSED), [e2e-llm-inference-service] errno.ENETUNREACH, [e2e-llm-inference-service] ) [e2e-llm-inference-service] if error.errno not in eConnRefused: [e2e-llm-inference-service] > raise error [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/websocket/_http.py:237: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] addrinfo_list = [(, , 6, '', ('3.22.17.175', 6443)), (,... '', ('3.148.74.118', 6443)), (, , 6, '', ('16.58.179.28', 6443))] [e2e-llm-inference-service] sockopt = [], timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def _open_socket(addrinfo_list, sockopt, timeout): [e2e-llm-inference-service] err = None [e2e-llm-inference-service] for addrinfo in addrinfo_list: [e2e-llm-inference-service] family, socktype, proto = addrinfo[:3] [e2e-llm-inference-service] sock = socket.socket(family, socktype, proto) [e2e-llm-inference-service] sock.settimeout(timeout) [e2e-llm-inference-service] for opts in DEFAULT_SOCKET_OPTION: [e2e-llm-inference-service] sock.setsockopt(*opts) [e2e-llm-inference-service] for opts in sockopt: [e2e-llm-inference-service] sock.setsockopt(*opts) [e2e-llm-inference-service] [e2e-llm-inference-service] address = addrinfo[4] [e2e-llm-inference-service] err = None [e2e-llm-inference-service] while not err: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > sock.connect(address) [e2e-llm-inference-service] E TimeoutError: [Errno 110] Connection timed out [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/websocket/_http.py:227: TimeoutError [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/api/v1/namespaces/e2e-test-llm-tls-resources-e4cad0c3/pods/tls-verification-test-kserve-656f77bc69-6b9c8/exec' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('name', 'tls-verification-test-kserve-656f77bc69-6b9c8'), ('namespace', 'e2e-test-llm-tls-resources-e4cad0c3')] [e2e-llm-inference-service] query_params = [('command', ['/bin/bash', '-c', "tr '\\0' '\\n' < /proc/1/cmdline"]), ('container', 'main'), ('stderr', True), ('stdin', False), ('stdout', True), ('tty', False)] [e2e-llm-inference-service] header_params = {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/36.0.3/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {} [e2e-llm-inference-service] response_types_map = {200: 'str', 401: None}, auth_settings = ['BearerToken'] [e2e-llm-inference-service] _return_http_data_only = True, collection_formats = {}, _preload_content = True [e2e-llm-inference-service] _request_timeout = None, _host = None, _request_auth = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_types_map=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None, [e2e-llm-inference-service] _request_auth=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth( [e2e-llm-inference-service] header_params, query_params, auth_settings, [e2e-llm-inference-service] request_auth=_request_auth) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:184: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] configuration = [e2e-llm-inference-service] _method = 'GET' [e2e-llm-inference-service] url = 'wss://ae09c812c6e404d9a80b724db9fbfb35-422a5c72c4122098.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/e2e-test-l...ommand=tr+%27%5C0%27+%27%5Cn%27+%3C+%2Fproc%2F1%2Fcmdline&container=main&stderr=True&stdin=False&stdout=True&tty=False' [e2e-llm-inference-service] kwargs = {'binary': False, 'query_params': [('command', ['/bin/bash', '-c', "tr '\\0' '\\n' < /proc/1/cmdline"]), ('container',...('tty', False)], 'headers': {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/36.0.3/python'}, 'post_params': [], ...} [e2e-llm-inference-service] [e2e-llm-inference-service] def websocket_call(configuration, _method, url, **kwargs): [e2e-llm-inference-service] """An internal function to be called in api-client when a websocket [e2e-llm-inference-service] connection is required. method, url, and kwargs are the parameters of [e2e-llm-inference-service] apiClient.request method.""" [e2e-llm-inference-service] [e2e-llm-inference-service] url = get_websocket_url(url, kwargs.get("query_params")) [e2e-llm-inference-service] headers = kwargs.get("headers") [e2e-llm-inference-service] _request_timeout = kwargs.get("_request_timeout", 60) [e2e-llm-inference-service] _preload_content = kwargs.get("_preload_content", True) [e2e-llm-inference-service] capture_all = kwargs.get("capture_all", True) [e2e-llm-inference-service] binary = kwargs.get('binary', False) [e2e-llm-inference-service] try: [e2e-llm-inference-service] client = WSClient(configuration, url, headers, capture_all, binary=binary) [e2e-llm-inference-service] if not _preload_content: [e2e-llm-inference-service] return client [e2e-llm-inference-service] client.run_forever(timeout=_request_timeout) [e2e-llm-inference-service] all = client.read_all() [e2e-llm-inference-service] if binary: [e2e-llm-inference-service] return WSResponse(data=all, status=200) [e2e-llm-inference-service] else: [e2e-llm-inference-service] return WSResponse(data='%s' % ''.join(all), status=200) [e2e-llm-inference-service] except (Exception, KeyboardInterrupt, SystemExit) as e: [e2e-llm-inference-service] > raise ApiException(status=0, reason=str(e)) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (0) [e2e-llm-inference-service] E Reason: [Errno 110] Connection timed out [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/stream/ws_client.py:597: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-fb-opt-125m'], system_base_refs=[], prompt='KServe is ... {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] service_name="tls-verification-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_tls_resources(test_case: TestCase): [e2e-llm-inference-service] """Verify that TLS-related resources (DestinationRules, cert secrets, service port) [e2e-llm-inference-service] are correctly present or absent based on the enableLLMInferenceServiceTLS flag.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] tls_enabled, _, _ = _get_tls_config() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] wait_for_model_response(kserve_client, test_case, test_case.wait_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] tls_enabled, tls_min_version, tls_cipher_suites = _get_tls_config() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] "LLMInferenceService TLS config: enabled=%s, min_version=%s, cipher_suites=%s", [e2e-llm-inference-service] tls_enabled, [e2e-llm-inference-service] tls_min_version, [e2e-llm-inference-service] tls_cipher_suites, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > wait_for( [e2e-llm-inference-service] lambda: _verify_tls_arguments( [e2e-llm-inference-service] service_name, [e2e-llm-inference-service] test_case.namespace, [e2e-llm-inference-service] tls_enabled, [e2e-llm-inference-service] tls_min_version, [e2e-llm-inference-service] tls_cipher_suites, [e2e-llm-inference-service] ), [e2e-llm-inference-service] timeout=test_case.wait_timeout, [e2e-llm-inference-service] interval=2.0, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:289: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = . at 0x7fe255dd8900> [e2e-llm-inference-service] timeout = 900, interval = 2.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] ^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1446: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] > lambda: _verify_tls_arguments( [e2e-llm-inference-service] service_name, [e2e-llm-inference-service] test_case.namespace, [e2e-llm-inference-service] tls_enabled, [e2e-llm-inference-service] tls_min_version, [e2e-llm-inference-service] tls_cipher_suites, [e2e-llm-inference-service] ), [e2e-llm-inference-service] timeout=test_case.wait_timeout, [e2e-llm-inference-service] interval=2.0, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:290: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = 'tls-verification-test' [e2e-llm-inference-service] namespace = 'e2e-test-llm-tls-resources-e4cad0c3', tls_enabled = True [e2e-llm-inference-service] tls_min_version = '', tls_cipher_suites = '' [e2e-llm-inference-service] [e2e-llm-inference-service] def _verify_tls_arguments( [e2e-llm-inference-service] service_name, namespace, tls_enabled, tls_min_version, tls_cipher_suites [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Assert that the TLS profile reaches Go components and the running vLLM.""" [e2e-llm-inference-service] > commands = _get_container_tls_state(namespace, service_name) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:391: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] namespace = 'e2e-test-llm-tls-resources-e4cad0c3' [e2e-llm-inference-service] service_name = 'tls-verification-test' [e2e-llm-inference-service] [e2e-llm-inference-service] def _get_container_tls_state(namespace, service_name): [e2e-llm-inference-service] """Return configured Go commands and the running vLLM command.""" [e2e-llm-inference-service] core_v1 = client.CoreV1Api() [e2e-llm-inference-service] pods = core_v1.list_namespaced_pod( [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] label_selector=( [e2e-llm-inference-service] f"app.kubernetes.io/part-of=llminferenceservice," [e2e-llm-inference-service] f"app.kubernetes.io/name={service_name}" [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] commands = {"epp": [], "sidecar": [], "vllm": []} [e2e-llm-inference-service] for pod in pods.items: [e2e-llm-inference-service] containers = [ [e2e-llm-inference-service] *(pod.spec.init_containers or []), [e2e-llm-inference-service] *(pod.spec.containers or []), [e2e-llm-inference-service] ] [e2e-llm-inference-service] for container in containers: [e2e-llm-inference-service] command = [*(container.command or []), *(container.args or [])] [e2e-llm-inference-service] if "/app/epp" in command: [e2e-llm-inference-service] commands["epp"].append(command) [e2e-llm-inference-service] elif "/app/pd-sidecar" in command: [e2e-llm-inference-service] commands["sidecar"].append(command) [e2e-llm-inference-service] elif any("vllm serve" in arg for arg in command): [e2e-llm-inference-service] try: [e2e-llm-inference-service] > runtime_state = k8s_stream( [e2e-llm-inference-service] core_v1.connect_get_namespaced_pod_exec, [e2e-llm-inference-service] pod.metadata.name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] container=container.name, [e2e-llm-inference-service] command=[ [e2e-llm-inference-service] "/bin/bash", [e2e-llm-inference-service] "-c", [e2e-llm-inference-service] "tr '\\0' '\\n' < /proc/1/cmdline", [e2e-llm-inference-service] ], [e2e-llm-inference-service] stderr=True, [e2e-llm-inference-service] stdin=False, [e2e-llm-inference-service] stdout=True, [e2e-llm-inference-service] tty=False, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:176: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] websocket_request = [e2e-llm-inference-service] force_kwargs = None [e2e-llm-inference-service] api_method = > [e2e-llm-inference-service] args = ('tls-verification-test-kserve-656f77bc69-6b9c8', 'e2e-test-llm-tls-resources-e4cad0c3') [e2e-llm-inference-service] kwargs = {'container': 'main', 'command': ['/bin/bash', '-c', "tr '\\0' '\\n' < /proc/1/cmdline"], 'stderr': True, 'stdin': False, ...} [e2e-llm-inference-service] [e2e-llm-inference-service] def _websocket_request(websocket_request, force_kwargs, api_method, *args, **kwargs): [e2e-llm-inference-service] """Override the ApiClient.request method with an alternative websocket based [e2e-llm-inference-service] method and call the supplied Kubernetes API method with that in place.""" [e2e-llm-inference-service] if force_kwargs: [e2e-llm-inference-service] for kwarg, value in force_kwargs.items(): [e2e-llm-inference-service] kwargs[kwarg] = value [e2e-llm-inference-service] api_client = api_method.__self__.api_client [e2e-llm-inference-service] # old generated code's api client has config. new ones has configuration [e2e-llm-inference-service] try: [e2e-llm-inference-service] configuration = api_client.configuration [e2e-llm-inference-service] except AttributeError: [e2e-llm-inference-service] configuration = api_client.config [e2e-llm-inference-service] prev_request = api_client.request [e2e-llm-inference-service] binary = kwargs.pop('binary', False) [e2e-llm-inference-service] try: [e2e-llm-inference-service] api_client.request = functools.partial(websocket_request, configuration, binary=binary) [e2e-llm-inference-service] > out = api_method(*args, **kwargs) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/stream/stream.py:36: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] name = 'tls-verification-test-kserve-656f77bc69-6b9c8' [e2e-llm-inference-service] namespace = 'e2e-test-llm-tls-resources-e4cad0c3' [e2e-llm-inference-service] kwargs = {'container': 'main', 'command': ['/bin/bash', '-c', "tr '\\0' '\\n' < /proc/1/cmdline"], 'stderr': True, 'stdin': False, ...} [e2e-llm-inference-service] [e2e-llm-inference-service] def connect_get_namespaced_pod_exec(self, name, namespace, **kwargs): # noqa: E501 [e2e-llm-inference-service] """connect_get_namespaced_pod_exec # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] connect GET requests to exec of Pod # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] [e2e-llm-inference-service] >>> thread = api.connect_get_namespaced_pod_exec(name, namespace, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param name: name of the PodExecOptions (required) [e2e-llm-inference-service] :type name: str [e2e-llm-inference-service] :param namespace: object name and auth scope, such as for teams and projects (required) [e2e-llm-inference-service] :type namespace: str [e2e-llm-inference-service] :param command: Command is the remote command to execute. argv array. Not executed within a shell. [e2e-llm-inference-service] :type command: str [e2e-llm-inference-service] :param container: Container in which to execute the command. Defaults to only container if there is only one container in the pod. [e2e-llm-inference-service] :type container: str [e2e-llm-inference-service] :param stderr: Redirect the standard error stream of the pod for this call. [e2e-llm-inference-service] :type stderr: bool [e2e-llm-inference-service] :param stdin: Redirect the standard input stream of the pod for this call. Defaults to false. [e2e-llm-inference-service] :type stdin: bool [e2e-llm-inference-service] :param stdout: Redirect the standard output stream of the pod for this call. [e2e-llm-inference-service] :type stdout: bool [e2e-llm-inference-service] :param tty: TTY if true indicates that a tty will be allocated for the exec call. Defaults to false. [e2e-llm-inference-service] :type tty: bool [e2e-llm-inference-service] :param async_req: Whether to execute the request asynchronously. [e2e-llm-inference-service] :type async_req: bool, optional [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :type _preload_content: bool, optional [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: Returns the result object. [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] :rtype: str [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501 [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/core_v1_api.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] name = 'tls-verification-test-kserve-656f77bc69-6b9c8' [e2e-llm-inference-service] namespace = 'e2e-test-llm-tls-resources-e4cad0c3' [e2e-llm-inference-service] kwargs = {'container': 'main', 'command': ['/bin/bash', '-c', "tr '\\0' '\\n' < /proc/1/cmdline"], 'stderr': True, 'stdin': False, ...} [e2e-llm-inference-service] [e2e-llm-inference-service] def connect_get_namespaced_pod_exec_with_http_info(self, name, namespace, **kwargs): # noqa: E501 [e2e-llm-inference-service] """connect_get_namespaced_pod_exec # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] connect GET requests to exec of Pod # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] [e2e-llm-inference-service] >>> thread = api.connect_get_namespaced_pod_exec_with_http_info(name, namespace, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param name: name of the PodExecOptions (required) [e2e-llm-inference-service] :type name: str [e2e-llm-inference-service] :param namespace: object name and auth scope, such as for teams and projects (required) [e2e-llm-inference-service] :type namespace: str [e2e-llm-inference-service] :param command: Command is the remote command to execute. argv array. Not executed within a shell. [e2e-llm-inference-service] :type command: str [e2e-llm-inference-service] :param container: Container in which to execute the command. Defaults to only container if there is only one container in the pod. [e2e-llm-inference-service] :type container: str [e2e-llm-inference-service] :param stderr: Redirect the standard error stream of the pod for this call. [e2e-llm-inference-service] :type stderr: bool [e2e-llm-inference-service] :param stdin: Redirect the standard input stream of the pod for this call. Defaults to false. [e2e-llm-inference-service] :type stdin: bool [e2e-llm-inference-service] :param stdout: Redirect the standard output stream of the pod for this call. [e2e-llm-inference-service] :type stdout: bool [e2e-llm-inference-service] :param tty: TTY if true indicates that a tty will be allocated for the exec call. Defaults to false. [e2e-llm-inference-service] :type tty: bool [e2e-llm-inference-service] :param async_req: Whether to execute the request asynchronously. [e2e-llm-inference-service] :type async_req: bool, optional [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :type _return_http_data_only: bool, optional [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :type _preload_content: bool, optional [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :param _request_auth: set to override the auth_settings for an a single [e2e-llm-inference-service] request; this effectively ignores the authentication [e2e-llm-inference-service] in the spec for a single request. [e2e-llm-inference-service] :type _request_auth: dict, optional [e2e-llm-inference-service] :type _content_type: string, optional: force content-type for the request [e2e-llm-inference-service] :return: Returns the result object. [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] :rtype: tuple(str, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'name', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'command', [e2e-llm-inference-service] 'container', [e2e-llm-inference-service] 'stderr', [e2e-llm-inference-service] 'stdin', [e2e-llm-inference-service] 'stdout', [e2e-llm-inference-service] 'tty' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout', [e2e-llm-inference-service] '_request_auth', [e2e-llm-inference-service] '_content_type', [e2e-llm-inference-service] '_headers' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method connect_get_namespaced_pod_exec" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and local_var_params.get('name') is None: # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `connect_get_namespaced_pod_exec`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and local_var_params.get('namespace') is None: # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `connect_get_namespaced_pod_exec`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] if local_var_params.get('command') is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('command', local_var_params['command'])) # noqa: E501 [e2e-llm-inference-service] if local_var_params.get('container') is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('container', local_var_params['container'])) # noqa: E501 [e2e-llm-inference-service] if local_var_params.get('stderr') is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('stderr', local_var_params['stderr'])) # noqa: E501 [e2e-llm-inference-service] if local_var_params.get('stdin') is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('stdin', local_var_params['stdin'])) # noqa: E501 [e2e-llm-inference-service] if local_var_params.get('stdout') is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('stdout', local_var_params['stdout'])) # noqa: E501 [e2e-llm-inference-service] if local_var_params.get('tty') is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('tty', local_var_params['tty'])) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = dict(local_var_params.get('_headers', {})) [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['*/*']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] response_types_map = { [e2e-llm-inference-service] 200: "str", [e2e-llm-inference-service] 401: None, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/api/v1/namespaces/{namespace}/pods/{name}/exec', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_types_map=response_types_map, [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats, [e2e-llm-inference-service] _request_auth=local_var_params.get('_request_auth')) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/core_v1_api.py:1335: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/api/v1/namespaces/{namespace}/pods/{name}/exec' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'name': 'tls-verification-test-kserve-656f77bc69-6b9c8', 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3'} [e2e-llm-inference-service] query_params = [('command', ['/bin/bash', '-c', "tr '\\0' '\\n' < /proc/1/cmdline"]), ('container', 'main'), ('stderr', True), ('stdin', False), ('stdout', True), ('tty', False)] [e2e-llm-inference-service] header_params = {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/36.0.3/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {} [e2e-llm-inference-service] response_types_map = {200: 'str', 401: None}, auth_settings = ['BearerToken'] [e2e-llm-inference-service] async_req = None, _return_http_data_only = True, collection_formats = {} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None, _host = None [e2e-llm-inference-service] _request_auth = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_types_map=None, auth_settings=None, [e2e-llm-inference-service] async_req=None, _return_http_data_only=None, [e2e-llm-inference-service] collection_formats=None,_preload_content=True, [e2e-llm-inference-service] _request_timeout=None, _host=None, _request_auth=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :param _request_auth: set to override the auth_settings for an a single [e2e-llm-inference-service] request; this effectively ignores the authentication [e2e-llm-inference-service] in the spec for a single request. [e2e-llm-inference-service] :type _request_token: dict, optional [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_types_map, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host, [e2e-llm-inference-service] _request_auth) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:374: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/api/v1/namespaces/e2e-test-llm-tls-resources-e4cad0c3/pods/tls-verification-test-kserve-656f77bc69-6b9c8/exec' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('name', 'tls-verification-test-kserve-656f77bc69-6b9c8'), ('namespace', 'e2e-test-llm-tls-resources-e4cad0c3')] [e2e-llm-inference-service] query_params = [('command', ['/bin/bash', '-c', "tr '\\0' '\\n' < /proc/1/cmdline"]), ('container', 'main'), ('stderr', True), ('stdin', False), ('stdout', True), ('tty', False)] [e2e-llm-inference-service] header_params = {'Accept': '*/*', 'User-Agent': 'OpenAPI-Generator/36.0.3/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {} [e2e-llm-inference-service] response_types_map = {200: 'str', 401: None}, auth_settings = ['BearerToken'] [e2e-llm-inference-service] _return_http_data_only = True, collection_formats = {}, _preload_content = True [e2e-llm-inference-service] _request_timeout = None, _host = None, _request_auth = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_types_map=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None, [e2e-llm-inference-service] _request_auth=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth( [e2e-llm-inference-service] header_params, query_params, auth_settings, [e2e-llm-inference-service] request_auth=_request_auth) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] except ApiException as e: [e2e-llm-inference-service] > e.body = e.body.decode('utf-8') if six.PY3 else e.body [e2e-llm-inference-service] ^^^^^^^^^^^^^ [e2e-llm-inference-service] E AttributeError: 'NoneType' object has no attribute 'decode' [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:190: AttributeError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2251 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-tls-resources-e4cad0c3 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2251 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2200 Checking LLMInferenceServiceConfig router-managed-tls-verification-86160844 in namespace e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2226 Resource not found, creating LLMInferenceServiceConfig router-managed-tls-verification-86160844 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2236 ✓ Successfully created LLMInferenceServiceConfig router-managed-tls-verification-86160844 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2200 Checking LLMInferenceServiceConfig workload-pd-cpu-tls-verificatio-cc747c7a in namespace e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2226 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-tls-verificatio-cc747c7a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2236 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-tls-verificatio-cc747c7a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2200 Checking LLMInferenceServiceConfig model-fb-opt-125m-tls-verificat-7c5fca28 in namespace e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2226 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-tls-verificat-7c5fca28 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2236 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-tls-verificat-7c5fca28 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_tls_resources] [2026-10-30T23:28:33.453662] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-fb-opt-125m'], system_base_refs=[], prompt='KServe is a', service_name='tls-verification-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7fe256453600>, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-tls-resources-e4cad0c3', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2251 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2251 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-10-30T23:28:33.554665] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-10-30T23:28:33.670019] end - ✅ in 0.115s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-10-30T23:28:33.670115] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1453 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1453 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'the DisaggregatedSet feature gate is disabled', 'reason': 'FeatureGateDisabled', 'severity': 'Info', 'status': 'False', 'type': 'DisaggregatedSetUsed'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route]', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Inference Pool e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1453 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'the DisaggregatedSet feature gate is disabled', 'reason': 'FeatureGateDisabled', 'severity': 'Info', 'status': 'False', 'type': 'DisaggregatedSetUsed'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1453 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'the DisaggregatedSet feature gate is disabled', 'reason': 'FeatureGateDisabled', 'severity': 'Info', 'status': 'False', 'type': 'DisaggregatedSetUsed'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-10-30T23:29:23Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-10-30T23:29:23Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1453 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'the DisaggregatedSet feature gate is disabled', 'reason': 'FeatureGateDisabled', 'severity': 'Info', 'status': 'False', 'type': 'DisaggregatedSetUsed'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-10-30T23:30:49Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-10-30T23:28:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-10-30T23:29:23Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-10-30T23:29:23Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-10-30T23:28:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-10-30T23:31:30.287340] end - ✅ in 176.617s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-10-30T23:31:30.287492] start - args=(, TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-fb-opt-125m'], system_base_refs=[], prompt='KServe is a', service_name='tls-verification-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7fe256453600>, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-tls-resources-e4cad0c3', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-10-30T23:31:30.287832] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-10-30T23:31:30.307802] end - ✅ in 0.020s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1316 Calling LLM service at http://a24be425ea59e4e2090e0045e75477c9-383088110.us-east-2.elb.amazonaws.com/e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1335 Model response is 200: {"id":"cmpl-edc30809-3e8d-45f9-b900-092a861cf69e","object":"text_completion","created":1793403093,"model":"facebook/opt-125m","choices":[{"index":0,"text":" web site for individuals interested in improving the health of their homes and communities. KServe allows individuals in","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":5,"total_tokens":25,"com [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_model_response] [2026-10-30T23:31:36.620540] end - ✅ in 6.333s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2251 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:test_llm_tls.py:282 LLMInferenceService TLS config: enabled=True, min_version=, cipher_suites= [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_tls:test_llm_tls.py:303 Failed TLS verification for tls-verification-test: 'NoneType' object has no attribute 'decode' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:274 # Diagnostics for 'tls-verification-test' in 'e2e-test-llm-tls-resources-e4cad0c3' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:275 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:278 # LLMInferenceService tls-verification-test [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:287 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:33Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:33Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:33Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/accelerator-type: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-confidential-bare-metal: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-omni-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-default: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-default-pd: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-modality: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-sglang-template: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-10-30T23:31:30Z' [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] resourceVersion: '67938' [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-tls-verification-86160844 [e2e-llm-inference-service] - name: workload-pd-cpu-tls-verificatio-cc747c7a [e2e-llm-inference-service] - name: model-fb-opt-125m-tls-verificat-7c5fca28 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a24be425ea59e4e2090e0045e75477c9-383088110.us-east-2.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a24be425ea59e4e2090e0045e75477c9-383088110.us-east-2.elb.amazonaws.com/e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a24be425ea59e4e2090e0045e75477c9-383088110.us-east-2.elb.amazonaws.com/publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/accelerator-type: cpu [e2e-llm-inference-service] serving.kserve.io/config-llm-confidential-bare-metal: kserve-config-llm-confidential-bare-metal [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-omni-template: kserve-config-llm-omni-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-default: kserve-config-llm-scheduler-eppconfig-default [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-default-pd: kserve-config-llm-scheduler-eppconfig-default-pd [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-modality: kserve-config-llm-scheduler-eppconfig-modality [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-sglang-template: kserve-config-sglang-template [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] message: the DisaggregatedSet feature gate is disabled [e2e-llm-inference-service] reason: FeatureGateDisabled [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: DisaggregatedSetUsed [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:31:30Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:31:30Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:31:30Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a24be425ea59e4e2090e0045e75477c9-383088110.us-east-2.elb.amazonaws.com/e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 None e2e-test-llm-tls-resources-e4cad0c3 Normal Scheduled Successfully assigned e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-656f77bc69-6b9c8 to ip-10-0-132-192.us-east-2.compute.internal [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:odh-v3.6.0" already present on machine and can be accessed by the pod [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Created Container created [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Started Container started [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937" already present on machine and can be accessed by the pod [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Created Container created [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Started Container started [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:53 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine and can be accessed by the pod [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:53 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Created Container created [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:53 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Normal Started Container started [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:30:39 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-132-192.us-east-2.compute.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8001/health": dial tcp 10.134.0.45:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:48 e2e-test-llm-tls-resources-e4cad0c3 replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-656f77bc69-6b9c8 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 None e2e-test-llm-tls-resources-e4cad0c3 Normal Scheduled Successfully assigned e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq to ip-10-0-138-251.us-east-2.compute.internal [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 multus Normal AddedInterface Add eth0 [10.132.0.90/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-138-251.us-east-2.compute.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937" already present on machine and can be accessed by the pod [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-138-251.us-east-2.compute.internal Normal Created Container created [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-138-251.us-east-2.compute.internal Normal Started Container started [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:53 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-138-251.us-east-2.compute.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine and can be accessed by the pod [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:53 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-138-251.us-east-2.compute.internal Normal Created Container created [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:53 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-138-251.us-east-2.compute.internal Normal Started Container started [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:31:19 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-138-251.us-east-2.compute.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.90:8000/health": dial tcp 10.132.0.90:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-prefill-577b5bd485-cjmrq [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-prefill-577b5bd485 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 None e2e-test-llm-tls-resources-e4cad0c3 Normal Scheduled Successfully assigned e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5 to ip-10-0-128-97.us-east-2.compute.internal [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 multus Normal AddedInterface Add eth0 [10.133.0.83/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-128-97.us-east-2.compute.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:odh-v3.6.0" already present on machine and can be accessed by the pod [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:50 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-128-97.us-east-2.compute.internal Normal Created Container created [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:51 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-128-97.us-east-2.compute.internal Normal Started Container started [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-router-scheduler-69f7c679f8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:48 e2e-test-llm-tls-resources-e4cad0c3 deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-656f77bc69 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 None e2e-test-llm-tls-resources-e4cad0c3 Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 None e2e-test-llm-tls-resources-e4cad0c3 Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:34 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:34 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:34 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:48 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:48 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:48 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:49 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-tls-resources-e4ca098b05fa5f51c9840356eab55a0cbfac [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:58 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:57 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-10-30 23:28:58 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:252 ### EndpointSlice tls-verification-test-epp-service-6hv9p (service=tls-verification-test-epp-service): [{'addresses': ['10.133.0.83'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5'}] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:252 ### EndpointSlice tls-verification-test-inference-pool-ip-e386aca5-dv6z5 (service=tls-verification-test-inference-pool-ip-e386aca5): [{'addresses': ['10.134.0.45'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-656f77bc69-6b9c8'}, {'addresses': ['10.132.0.90'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-prefill-577b5bd485-cjmrq'}] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:252 ### EndpointSlice tls-verification-test-kserve-workload-svc-5nhfn (service=tls-verification-test-kserve-workload-svc): [{'addresses': ['10.134.0.45'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-656f77bc69-6b9c8'}, {'addresses': ['10.132.0.90'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-prefill-577b5bd485-cjmrq'}] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:172 ### Pod tls-verification-test-kserve-656f77bc69-6b9c8 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b'Flag --secure-proxy has been deprecated, use --secure-serving instead\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:49.973758887Z","body":"Proxy starting","service.name":"llm-d-router-disagg-sidecar","Built on":"","From Git SHA":"unknown","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:49.973816847Z","body":"Proxy configuration","service.name":"llm-d-router-disagg-sidecar","config":"{\\"Port\\":\\"8000\\",\\"KVConnector\\":\\"nixlv2\\",\\"ECConnector\\":\\"\\",\\"DataParallelSize\\":1,\\"MaxIdleConnsPerHost\\":1024,\\"EnablePrefillerSampling\\":false,\\"PrefillMaxRetries\\":0,\\"PrefillRetryBackoff\\":200000000,\\"UseTLSForPrefiller\\":true,\\"UseTLSForDecoder\\":true,\\"UseTLSForEncoder\\":false,\\"InsecureSkipVerifyForPrefiller\\":false,\\"InsecureSkipVerifyForEncoder\\":false,\\"InsecureSkipVerifyForDecoder\\":false,\\"SecureServing\\":true,\\"CertPath\\":\\"/var/run/kserve/tls\\",\\"TLSMinVersion\\":0,\\"TLSCipherSuites\\":null,\\"MetricsPort\\":0,\\"MetricsCertDir\\":\\"\\",\\"MooncakeBootstrapPort\\":8998,\\"P2PConnectorPort\\":7777,\\"EnableP2PPull\\":false,\\"EnableSSRFProtection\\":true,\\"InferencePoolNamespace\\":\\"e2e-test-llm-tls-resources-e4cad0c3\\",\\"InferencePoolName\\":\\"tls-verification-test-inference-pool\\",\\"PoolGroup\\":\\"inference.networking.k8s.io\\",\\"DecodeChunkSize\\":0,\\"Tracing\\":false,\\"EnableToolCallingTelemetry\\":false,\\"MoRIIOWriteMode\\":false,\\"MoRIIODecodeNotifyPort\\":61005,\\"MoRIIODecodeHandshakePort\\":6301,\\"MoRIIODecodePodIP\\":\\"\\",\\"MoRIIOParallelDispatch\\":false,\\"MoRIIOParallelDecodeWaitTimeout\\":30000000000,\\"MoRIIOPrefillHandshakePort\\":6301,\\"MoRIIOPrefillNotifyPort\\":61005,\\"MoRIIOTPSize\\":1,\\"MoRIIODPSize\\":1,\\"MoRIIORemoteHosts\\":null,\\"MoRIIODPSizeLocal\\":0,\\"MoRIIODecodeHosts\\":null,\\"MoRIIORemoteHostSpecs\\":null,\\"MoRIIODecodeHostSpecs\\":null,\\"MoRIIODecodePodIPSpec\\":\\"\\",\\"DecoderURL\\":\\"https://localhost:8001\\"}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:49.974277201Z","logger":"allowlist-validator","body":"starting SSRF protection allowlist validator","service.name":"llm-d-router-disagg-sidecar","namespace":"e2e-test-llm-tls-resources-e4cad0c3","poolName":"tls-verification-test-inference-pool","gvr":"inference.networking.k8s.io/v1, Resource=inferencepools","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:49.995565944Z","logger":"allowlist-validator","body":"InferencePool added","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:50.008556457Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":0,"targets":{},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:50.008654737Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":0,"targets":{},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:50.075164797Z","logger":"allowlist-validator","body":"allowlist validator started successfully","service.name":"llm-d-router-disagg-sidecar","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:50.078504446Z","logger":"proxy server on port 8000","body":"server TLS","service.name":"llm-d-router-disagg-sidecar","tls":true,"cert_path":"/var/run/kserve/tls","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:50.078560346Z","logger":"proxy server on port 8000","body":"starting","service.name":"llm-d-router-disagg-sidecar","addr":"[::]:8000","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:50.079426755Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":0,"targets":{},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:50.511470353Z","log [e2e-llm-inference-service] ger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":2,"targets":{"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:50.74171765Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.524246054Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:53.532135527Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:53.742220754Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:54.539410341Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:54.748199023Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:58.011813228Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:58.026847521Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:58.026921819Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:59.31902481Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:29:19.995790456Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d- [e2e-llm-inference-service] router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:29:20.010547408Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:29:20.010606144Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:29:49.995928724Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:29:50.007945921Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:29:50.007995968Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:30:19.996978218Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:30:20.010011935Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:30:20.01007624Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:30:49.324004085Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:30:49.352831692Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:30:49.997435647Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:30:50.010766826Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount": [e2e-llm-inference-service] 4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:30:50.01082562Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:31:19.997507254Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:31:20.010501106Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:31:20.010557653Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:31:29.518093639Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:31:29.542100504Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:31:49.997537849Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:31:50.010083256Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:31:50.01013734Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:32:19.997962756Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:32:20.010469519Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:32:20.010525309Z","logger": [e2e-llm-inference-service] "allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:32:49.998508656Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:32:50.011100811Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:32:50.011164273Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:33:19.999221366Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:33:20.011944425Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:33:20.012024385Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.90:8000":{},"10.134.0.45:8000":{},"tls-verification-test-kserve-656f77bc69-6b9c8:8000":{},"tls-verification-test-kserve-prefill-577b5bd485-cjmrq:8000":{}},"severity_number":9}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b"2026-10-30 23:28:51.015 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://example-models/facebook/opt-125m', '/mnt/models')]\n2026-10-30 23:28:51.015 1 storage.initializer INFO [kserve_storage.py:download():608] Copying contents of s3://example-models/facebook/opt-125m to local\n2026-10-30 23:28:51.031 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():905] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpji7h5sdt.crt\n2026-10-30 23:28:51.497 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/.gitignore (1 bytes)\n2026-10-30 23:28:51.497 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/.gitattributes.lock\n2026-10-30 23:28:51.497 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/.gitattributes.metadata (101 bytes)\n2026-10-30 23:28:51.497 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/LICENSE.md.lock\n2026-10-30 23:28:51.497 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/LICENSE.md.metadata (100 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/README.md.lock\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/README.md.metadata (100 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/config.json.lock\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/config.json.metadata (101 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/generation_config.json.lock\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/generation_config.json.metadata (101 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/merges.txt.lock\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/merges.txt.metadata (100 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.lock\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.metadata (125 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.lock\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.metadata (101 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.lock\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.metadata (100 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer DEBUG [kserve_ [e2e-llm-inference-service] storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/vocab.json.lock\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/vocab.json.metadata (101 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.gitattributes (1173 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/LICENSE.md (11117 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/README.md (7099 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/config.json (651 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/generation_config.json (137 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/merges.txt (456318 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/pytorch_model.bin (250540281 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/special_tokens_map.json (441 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/tokenizer_config.json (685 bytes)\n2026-10-30 23:28:51.498 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/vocab.json (898822 bytes)\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/.gitignore\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/.gitattributes.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/LICENSE.md.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/README.md.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/config.json.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/generation_config.json.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/merges.txt.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/vocab.json.metadata\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.gitattributes\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/LICENSE.md\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.metadata\n2026-1 [e2e-llm-inference-service] 0-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/config.json\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/generation_config.json\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/README.md\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/tokenizer_config.json\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/special_tokens_map.json\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/merges.txt\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/vocab.json\n2026-10-30 23:28:52.824 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/pytorch_model.bin\n2026-10-30 23:28:52.825 1 storage.initializer INFO [kserve_storage.py:download():699] Successfully copied s3://example-models/facebook/opt-125m to /mnt/models\n2026-10-30 23:28:52.825 1 storage.initializer INFO [kserve_storage.py:download():700] Model downloaded in 1.8102867950001382 seconds.\n" [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b'DEBUG 10-30 23:29:53 [platforms/__init__.py:165] Checking if CPU platform is available.\nDEBUG 10-30 23:29:53 [platforms/__init__.py:169] Confirmed CPU platform is available because vLLM is built with CPU.\nDEBUG 10-30 23:29:53 [platforms/__init__.py:165] Checking if CPU platform is available.\nDEBUG 10-30 23:29:53 [platforms/__init__.py:169] Confirmed CPU platform is available because vLLM is built with CPU.\nDEBUG 10-30 23:29:53 [platforms/__init__.py:247] Automatically detected platform cpu.\n(APIServer pid=1) DEBUG 10-30 23:29:56 [v1/engine/utils.py:1042] Waiting for 1 local, 0 remote core engine proc(s) to connect.\nINFO 10-30 23:29:57 [triton_utils/importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.\nINFO 10-30 23:29:57 [triton_utils/importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available.\nDEBUG 10-30 23:29:57 [utils/flashinfer.py:52] FlashInfer unavailable since package was not found\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [v1/engine/core.py:1018] Waiting for init message from front-end.\n(APIServer pid=1) DEBUG 10-30 23:30:02 [v1/engine/utils.py:1158] HELLO from local core engine process 0.\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [v1/engine/core.py:1029] Received init message: EngineHandshakeMetadata(addresses=EngineZmqAddresses(inputs=[\'ipc:///tmp/365075da-f42d-4d71-8ff8-83edb878e49a\'], outputs=[\'ipc:///tmp/65e6107d-1efe-48d4-9009-f5bca59de069\'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None), parallel_config={})\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [v1/engine/core.py:826] Has DP Coordinator: False, stats publish address: None\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [plugins/__init__.py:44] Available plugins for group vllm.general_plugins:\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [plugins/__init__.py:46] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [plugins/__init__.py:46] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [plugins/__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.\n(EngineCore pid=72) INFO 10-30 23:30:02 [v1/engine/core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model=\'/mnt/models\', speculative_config=None, tokenizer=\'/mnt/models\', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend=\'auto\', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=\'\', reasoning_parser_plugin=\'\', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=facebook/opt-125m, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={\'mode\': , \'debug_dump_path\': None, \'cache_dir\': \'\', \'compile_cache_save_format\': \'binary\', \'backend\': \'inductor\', \'custom_ops\': [\'none\'], \'splitting_ops\': [], \'compile_mm_encoder\': False, \'cudagraph_mm_encoder\': False, \'encoder_cudagraph_token_budgets\': [e2e-llm-inference-service] [], \'encoder_cudagraph_max_images_per_batch\': 0, \'compile_sizes\': None, \'compile_ranges_endpoints\': [2048], \'inductor_compile_config\': {\'enable_auto_functionalized_v2\': False, \'size_asserts\': False, \'alignment_asserts\': True, \'scalar_asserts\': True, \'dce\': True, \'nan_asserts\': False, \'epilogue_fusion\': True, \'cpp.dynamic_threads\': True}, \'inductor_passes\': {}, \'cudagraph_mode\': , \'cudagraph_num_of_warmups\': 0, \'cudagraph_capture_sizes\': [], \'cudagraph_copy_inputs\': False, \'cudagraph_specialize_lora\': True, \'use_inductor_graph_partition\': False, \'pass_config\': {\'fuse_norm_quant\': False, \'fuse_act_quant\': False, \'fuse_attn_quant\': False, \'enable_sp\': False, \'fuse_gemm_comms\': False, \'fuse_allreduce_rms\': False}, \'max_cudagraph_capture_size\': None, \'dynamic_shapes_config\': {\'type\': , \'evaluate_guards\': False, \'assume_32_bit_indexing\': False}, \'local_cache_dir\': None, \'fast_moe_cold_start\': True, \'static_all_moe_layers\': []}\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'input_ids\', \'positions\', \'intermediate_tensors\', \'inputs_embeds\']\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'input_ids\', \'positions\', \'hidden_states\', \'input_embeds\']\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'input_ids\', \'positions\', \'intermediate_tensors\', \'inputs_embeds\']\n(EngineCore pid=72) DEBUG 10-30 23:30:02 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'num_tokens_no_spec\', \'token_ids_gpu\', \'combined_mask\']\n(EngineCore pid=72) INFO 10-30 23:30:03 [v1/worker/cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)]\n[W1030 23:30:03.248099282 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env)\n[W1030 23:30:03.248129455 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env)\n(EngineCore pid=72) INFO 10-30 23:30:03 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 72:\r\n(EngineCore pid=72) INFO 10-30 23:30:03 [v1/worker/cpu_worker.py:109] \tOMP tid: 72, core 4\r\n(EngineCore pid=72) INFO 10-30 23:30:03 [v1/worker/cpu_worker.py:109] \tOMP tid: 89, core 5\r\n(EngineCore pid=72) INFO 10-30 23:30:03 [v1/worker/cpu_worker.py:109] \tOMP tid: 90, core 6\r\n(EngineCore pid=72) INFO 10-30 23:30:03 [v1/worker/cpu_worker.py:109] \tOMP tid: 91, core 7\r\n(EngineCore pid=72) INFO 10-30 23:30:03 [v1/worker/cpu_worker.py:109] \n(EngineCore pid=72) DEBUG 10-30 23:30:03 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.134.0.45:58849 backend=gloo\n(EngineCore pid=72) INFO 10-30 23:30:03 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.134.0.45:58849 backend=gloo\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n(EngineCore pid=72) DEBUG 10-30 23:30:03 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : [e2e-llm-inference-service] 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n(EngineCore pid=72) INFO 10-30 23:30:03 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A\n(EngineCore pid=72) DEBUG 10-30 23:30:03 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors).\n(EngineCore pid=72) DEBUG 10-30 23:30:03 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading).\n(EngineCore pid=72) INFO 10-30 23:30:03 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models...\n(EngineCore pid=72) DEBUG 10-30 23:30:03 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'input_ids\', \'positions\', \'intermediate_tensors\', \'inputs_embeds\']\n(EngineCore pid=72) DEBUG 10-30 23:30:03 [config/compilation.py:1194] enabled custom ops: Counter()\n(EngineCore pid=72) DEBUG 10-30 23:30:03 [config/compilation.py:1195] disabled custom ops: Counter({\'vocab_parallel_embedding\': 1, \'logits_processor\': 1})\n(EngineCore pid=72) DEBUG 10-30 23:30:03 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ...\n(EngineCore pid=72) \rLoading pt checkpoint shards: 0% Completed | 0/1 [00:00\n(APIServer pid=1) DEBUG 10-30 23:30:12 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(APIServer pid=1) DEBUG 10-30 23:30:22 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(APIServer pid=1) DEBUG 10-30 23:30:32 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(EngineCore pid=72) INFO 10-30 23:30:39 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/61c987e4d85cc4abdc6df0e8413bd783d9891939692f8a3336d2fea1085ff068/rank_0_0/model\n(EngineCore pid=72) INFO 10-30 23:30:40 [compilation/monitor [e2e-llm-inference-service] .py:76] Initial profiling/warmup run took 1.13 s\n(EngineCore pid=72) INFO 10-30 23:30:40 [v1/worker/cpu_model_runner.py:92] Warming up done.\n(EngineCore pid=72) INFO 10-30 23:30:40 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 36.81 seconds\n(EngineCore pid=72) DEBUG 10-30 23:30:40 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode=\'hf\'\n(EngineCore pid=72) DEBUG 10-30 23:30:40 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1\n(EngineCore pid=72) INFO 10-30 23:30:40 [config/vllm.py:790] Asynchronous scheduling is disabled.\n(APIServer pid=1) DEBUG 10-30 23:30:40 [v1/engine/utils.py:1158] READY from local core engine process 0.\n(EngineCore pid=72) WARNING 10-30 23:30:40 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.\n(EngineCore pid=72) DEBUG 10-30 23:30:40 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) DEBUG 10-30 23:30:40 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227\n(EngineCore pid=72) DEBUG 10-30 23:30:40 [v1/engine/core.py:1158] EngineCore waiting for work.\n(EngineCore pid=72) DEBUG 10-30 23:30:40 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) INFO 10-30 23:30:40 [entrypoints/openai/api_server.py:590] Supported tasks: [\'generate\']\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/base.py:197] Warming up chat template processing...\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] Traceback (most recent call last):\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools)\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] raise ValueError(\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:141] There is no chat template fallback for /mnt/models\n(APIServer pid=1) INFO 10-30 23:30:41 [renderers/hf.py:314] Detected the chat template content format to be \'string\'. You can set `--chat-template-content-format` to override this.\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] Traceback (most recent call last):\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools)\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template\r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] raise ValueError(\ [e2e-llm-inference-service] r\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating\n(APIServer pid=1) DEBUG 10-30 23:30:41 [renderers/base.py:205] This model does not support chat template.\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:37] Available routes are:\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: HEAD, GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /docs, Methods: HEAD, GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /redoc, Methods: HEAD, GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /load, Methods: GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /version, Methods: GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /health, Methods: GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /ping, Methods: GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /ping, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST\n(APIServer pid=1) INFO: Started server process [1]\n(APIServer pid=1) INFO: Waiting for application startup.\n(APIServer pid=1) INFO 10-30 23:30:41 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: [\'/var/run/kserve/tls/tls.key\', \'/var/run/kserve/tls/tls.crt\']\n(APIServer pid=1) INFO: Application startup complete.\n(APIServer pid=1) DEBUG 10-30 23:30:49 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:30:51 [v1/metrics/loggers.py:259] Engine 000: Avg prompt [e2e-llm-inference-service] throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:31:01 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:31:11 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:31:21 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:31:31 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) WARNING 10-30 23:31:33 [tracing/utils.py:72] Received a request with trace context but tracing is disabled\n(APIServer pid=1) DEBUG 10-30 23:31:33 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors).\n(EngineCore pid=72) DEBUG 10-30 23:31:33 [v1/engine/core.py:1170] EngineCore loop active.\n(EngineCore pid=72) DEBUG 10-30 23:31:33 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=5, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:33 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:35 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:35 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:35 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:35 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batc [e2e-llm-inference-service] h_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mo [e2e-llm-inference-service] de: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:36 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) INFO: 10.132.0.84:0 - "POST /v1/completions HTTP/1.1" 200 OK\n(APIServer pid=1) INFO 10-30 23:31:41 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.5 tokens/s, Avg generation throughput: 2.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) INFO 10-30 23:31:51 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:01 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:11 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:21 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:31 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:41 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:51 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:01 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:11 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 r [e2e-llm-inference-service] eqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:21 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:31 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:41 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:172 ### Pod tls-verification-test-kserve-prefill-577b5bd485-cjmrq (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b"2026-10-30 23:28:50.808 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://example-models/facebook/opt-125m', '/mnt/models')]\n2026-10-30 23:28:50.809 1 storage.initializer INFO [kserve_storage.py:download():608] Copying contents of s3://example-models/facebook/opt-125m to local\n2026-10-30 23:28:50.840 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():905] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmphc1q46jm.crt\n2026-10-30 23:28:51.353 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/.gitignore (1 bytes)\n2026-10-30 23:28:51.353 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/.gitattributes.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/.gitattributes.metadata (101 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/LICENSE.md.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/LICENSE.md.metadata (100 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/README.md.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/README.md.metadata (100 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/config.json.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/config.json.metadata (101 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/generation_config.json.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/generation_config.json.metadata (101 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/merges.txt.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/merges.txt.metadata (100 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.metadata (125 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.metadata (101 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.metadata (100 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer DEBUG [kserve_ [e2e-llm-inference-service] storage.py:_download_s3():991] Skipping: facebook/opt-125m/.cache/huggingface/download/vocab.json.lock\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.cache/huggingface/download/vocab.json.metadata (101 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/.gitattributes (1173 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/LICENSE.md (11117 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/README.md (7099 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/config.json (651 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/generation_config.json (137 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/merges.txt (456318 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/pytorch_model.bin (250540281 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/special_tokens_map.json (441 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/tokenizer_config.json (685 bytes)\n2026-10-30 23:28:51.354 1 storage.initializer INFO [kserve_storage.py:_download_s3():994] Found S3 object: facebook/opt-125m/vocab.json (898822 bytes)\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/.gitignore\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/LICENSE.md.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/README.md.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/.gitattributes.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/generation_config.json.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/config.json.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/merges.txt.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/vocab.json.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.metadata\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/.gitattributes\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/config.json\n2026- [e2e-llm-inference-service] 10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/LICENSE.md\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/README.md\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/generation_config.json\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/special_tokens_map.json\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/merges.txt\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/tokenizer_config.json\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/vocab.json\n2026-10-30 23:28:52.849 1 storage.initializer INFO [kserve_storage.py:_download_s3():1046] Downloaded object facebook/opt-125m/pytorch_model.bin\n2026-10-30 23:28:52.850 1 storage.initializer INFO [kserve_storage.py:download():699] Successfully copied s3://example-models/facebook/opt-125m to /mnt/models\n2026-10-30 23:28:52.851 1 storage.initializer INFO [kserve_storage.py:download():700] Model downloaded in 2.041552958000466 seconds.\n" [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b'(APIServer pid=1) DEBUG 10-30 23:29:48 [engine/arg_utils.py:2116] Enabling chunked prefill by default\n(APIServer pid=1) DEBUG 10-30 23:29:48 [engine/arg_utils.py:2146] Enabling prefix caching by default\n(APIServer pid=1) DEBUG 10-30 23:29:48 [engine/arg_utils.py:2232] Defaulting max_num_batched_tokens to 2048 for OPENAI_API_SERVER usage context.\n(APIServer pid=1) DEBUG 10-30 23:29:48 [engine/arg_utils.py:2242] Defaulting max_num_seqs to 128 for OPENAI_API_SERVER usage context.\n(APIServer pid=1) DEBUG 10-30 23:29:48 [config/parallel.py:811] Disabled the custom all-reduce kernel because it is not supported on current platform.\n(APIServer pid=1) DEBUG 10-30 23:29:48 [config/parallel.py:811] Disabled the custom all-reduce kernel because it is not supported on current platform.\n(APIServer pid=1) INFO 10-30 23:29:48 [config/vllm.py:790] Asynchronous scheduling is enabled.\n(APIServer pid=1) DEBUG 10-30 23:29:48 [plugins/__init__.py:36] No plugins for group vllm.stat_logger_plugins found.\n(APIServer pid=1) DEBUG 10-30 23:29:48 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode=\'hf\'\n(APIServer pid=1) DEBUG 10-30 23:29:48 [renderers/registry.py:57] Loading HfRenderer for renderer_mode=\'hf\'\n(APIServer pid=1) DEBUG 10-30 23:29:48 [plugins/io_processors/__init__.py:36] No IOProcessor plugins requested by the model\nDEBUG 10-30 23:29:56 [plugins/__init__.py:36] No plugins for group vllm.platform_plugins found.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:37] Checking if TPU platform is available.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:56] TPU platform is not available because: No module named \'libtpu\'\nDEBUG 10-30 23:29:56 [platforms/__init__.py:62] Checking if CUDA platform is available.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:89] Exception happens when checking CUDA platform: NVML Shared Library Not Found\nDEBUG 10-30 23:29:56 [platforms/__init__.py:106] CUDA platform is not available because: NVML Shared Library Not Found\nDEBUG 10-30 23:29:56 [platforms/__init__.py:113] Checking if ROCm platform is available.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:127] ROCm platform is not available because: No module named \'amdsmi\'\nDEBUG 10-30 23:29:56 [platforms/__init__.py:134] Checking if XPU platform is available.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:165] Checking if CPU platform is available.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:169] Confirmed CPU platform is available because vLLM is built with CPU.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:165] Checking if CPU platform is available.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:169] Confirmed CPU platform is available because vLLM is built with CPU.\nDEBUG 10-30 23:29:56 [platforms/__init__.py:247] Automatically detected platform cpu.\n(APIServer pid=1) DEBUG 10-30 23:29:58 [v1/engine/utils.py:1042] Waiting for 1 local, 0 remote core engine proc(s) to connect.\nINFO 10-30 23:30:00 [triton_utils/importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.\nINFO 10-30 23:30:00 [triton_utils/importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available.\nDEBUG 10-30 23:30:00 [utils/flashinfer.py:52] FlashInfer unavailable since package was not found\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [v1/engine/core.py:1018] Waiting for init message from front-end.\n(APIServer pid=1) DEBUG 10-30 23:30:05 [v1/engine/utils.py:1158] HELLO from local core engine process 0.\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [v1/engine/core.py:1029] Received init message: EngineHandshakeMetadata(addresses=EngineZmqAddresses(inputs=[\'ipc:///tmp/bce0c3e7-d4d7-4731-acb6-b29a04de8d0b\'], outputs=[\'ipc:///tmp/e9b19ab2-c531-4235-ac80-df6a29dd9e75\'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None), parallel_config={})\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [v1/engine/core.py:826] Has DP Coordinator: False, stats publish address: None\n(EngineCor [e2e-llm-inference-service] e pid=72) DEBUG 10-30 23:30:05 [plugins/__init__.py:44] Available plugins for group vllm.general_plugins:\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [plugins/__init__.py:46] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [plugins/__init__.py:46] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [plugins/__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/engine/core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model=\'/mnt/models\', speculative_config=None, tokenizer=\'/mnt/models\', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend=\'auto\', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=\'\', reasoning_parser_plugin=\'\', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=facebook/opt-125m, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={\'mode\': , \'debug_dump_path\': None, \'cache_dir\': \'\', \'compile_cache_save_format\': \'binary\', \'backend\': \'inductor\', \'custom_ops\': [\'none\'], \'splitting_ops\': [], \'compile_mm_encoder\': False, \'cudagraph_mm_encoder\': False, \'encoder_cudagraph_token_budgets\': [], \'encoder_cudagraph_max_images_per_batch\': 0, \'compile_sizes\': None, \'compile_ranges_endpoints\': [2048], \'inductor_compile_config\': {\'enable_auto_functionalized_v2\': False, \'size_asserts\': False, \'alignment_asserts\': True, \'scalar_asserts\': True, \'dce\': True, \'nan_asserts\': False, \'epilogue_fusion\': True, \'cpp.dynamic_threads\': True}, \'inductor_passes\': {}, \'cudagraph_mode\': , \'cudagraph_num_of_warmups\': 0, \'cudagraph_capture_sizes\': [], \'cudagraph_copy_inputs\': False, \'cudagraph_specialize_lora\': True, \'use_inductor_graph_partition\': False, \'pass_config\': {\'fuse_norm_quant\': False, \'fuse_act_quant\': False, \'fuse_attn_quant\': False, \'enable_sp\': False, \'fuse_gemm_comms\': False, \'fuse_allreduce_rms\': False}, \'max_cudagraph_capture_size\': None, \'dynamic_shapes_config\': {\'type\': , \'evaluate_guards\': False, \'assume_32_bit_indexing\': False}, \'local_cache_dir\': None, \'fast_moe_cold_start\': True, \'static_all_moe_layers\': []}\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'input_ids\', \'positions\', \'intermediate_tensors\', \'inputs_embeds\']\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'input_ids\', \'positions\', \'hidden_states\', \'input_embeds\']\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'input_ids\', \'posi [e2e-llm-inference-service] tions\', \'intermediate_tensors\', \'inputs_embeds\']\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'num_tokens_no_spec\', \'token_ids_gpu\', \'combined_mask\']\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/worker/cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)]\n[W1030 23:30:05.265023529 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env)\n[W1030 23:30:05.265048817 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env)\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 72:\r\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/worker/cpu_worker.py:109] \tOMP tid: 72, core 4\r\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/worker/cpu_worker.py:109] \tOMP tid: 89, core 5\r\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/worker/cpu_worker.py:109] \tOMP tid: 90, core 6\r\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/worker/cpu_worker.py:109] \tOMP tid: 91, core 7\r\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/worker/cpu_worker.py:109] \n(EngineCore pid=72) DEBUG 10-30 23:30:05 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.90:59991 backend=gloo\n(EngineCore pid=72) INFO 10-30 23:30:05 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.90:59991 backend=gloo\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0\n(EngineCore pid=72) INFO 10-30 23:30:05 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors).\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading).\n(EngineCore pid=72) INFO 10-30 23:30:05 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models...\n(EngineCore pid=72) DEBUG 10-30 23:30:05 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : [\'input_ids\', \'positions\', \'intermediate_tensors\', \'inputs_embeds\']\n(EngineCore pid=72) DEBUG 10-30 23:30:06 [config/compilation.py:1194] enabled custom ops: Counter()\n(EngineCore pid=72) DEBUG 10-30 23:30:06 [config/compilation.py:1195] disabled custom ops: Counter({\'vocab_parallel_embedding\': [e2e-llm-inference-service] 1, \'logits_processor\': 1})\n(EngineCore pid=72) DEBUG 10-30 23:30:06 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ...\n(EngineCore pid=72) \rLoading pt checkpoint shards: 0% Completed | 0/1 [00:00\n(APIServer pid=1) DEBUG 10-30 23:30:15 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(APIServer pid=1) DEBUG 10-30 23:30:25 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(APIServer pid=1) DEBUG 10-30 23:30:35 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(APIServer pid=1) DEBUG 10-30 23:30:45 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(APIServer pid=1) DEBUG 10-30 23:30:55 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(APIServer pid=1) DEBUG 10-30 23:31:05 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(APIServer pid=1) DEBUG 10-30 23:31:15 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start.\n(EngineCore pid=72) INFO 10-30 23:31:22 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/e1dca4ab5ed857f724fdea646191aa066492e66db474e8f2b86ec5611de93645/rank_0_0/model\n(EngineCore pid=72) INFO 10-30 23:31:22 [compilation/monitor.py:76] Initial profiling/warmup run took 0.95 s\n(EngineCore pid=72) INFO 10-30 23:31:22 [v1/worker/cpu_model_runner.py:92] Warming up done.\n(EngineCore pid=72) INFO 10-30 23:31:22 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 76.35 seconds\n(EngineCore pid=72) DEBUG 10-30 23:31:22 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode=\'hf\'\n(EngineCore pid=72) DEBUG 10-30 23:31:23 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1\n(EngineCore pid=72) INFO 10-30 23:31:23 [config/vllm.py:790] Asynchronous scheduling is disabled.\n(APIServer pid=1) DEBUG 10-30 23:31:23 [v1/engine/utils.py:1158] READY from local core engine process 0.\n(EngineCore pid=72) WARNING 10-30 23:31:23 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.\n(EngineCore pid=72) DEBUG 10-30 23:31:23 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) DEBUG 10-30 23:31:23 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227\n(EngineCore pid=72) DEBUG 10-30 23:31:23 [v1/engine/core.py:1158] EngineCore waiting for work.\n(EngineCore pid=72) DEBUG 10-30 2 [e2e-llm-inference-service] 3:31:23 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) INFO 10-30 23:31:23 [entrypoints/openai/api_server.py:590] Supported tasks: [\'generate\']\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/base.py:197] Warming up chat template processing...\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] Traceback (most recent call last):\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools)\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] raise ValueError(\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:141] There is no chat template fallback for /mnt/models\n(APIServer pid=1) INFO 10-30 23:31:24 [renderers/hf.py:314] Detected the chat template content format to be \'string\'. You can set `--chat-template-content-format` to override this.\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] Traceback (most recent call last):\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools)\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] raise ValueError(\r\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating\n(APIServer pid=1) DEBUG 10-30 23:31:24 [renderers/base.py:205] This model does not support chat template.\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8000\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:37] Available routes are:\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: GET, HEAD\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /docs, Methods: GET, HEAD\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /redoc, Methods: GET, HEAD\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route [e2e-llm-inference-service] : /detokenize, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /load, Methods: GET\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /version, Methods: GET\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /health, Methods: GET\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /ping, Methods: GET\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /ping, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST\n(APIServer pid=1) INFO: Started server process [1]\n(APIServer pid=1) INFO: Waiting for application startup.\n(APIServer pid=1) INFO 10-30 23:31:24 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: [\'/var/run/kserve/tls/tls.key\', \'/var/run/kserve/tls/tls.crt\']\n(APIServer pid=1) INFO: Application startup complete.\n(APIServer pid=1) DEBUG 10-30 23:31:29 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:31:29 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) WARNING 10-30 23:31:30 [tracing/utils.py:72] Received a request with trace context but tracing is disabled\n(APIServer pid=1) DEBUG 10-30 23:31:30 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors).\n(EngineCore pid=72) WARNING 10-30 23:31:30 [v1/engine/core.py:319] Got kv_transfer_params, but no KVConnector found. Disabling KVTransfer for this request.\n(EngineCore pid=72) DEBUG 10-30 23:31:30 [v1/engine/core.py:1170] EngineCore loop active.\n(EngineCore pid=72) DEBUG 10-30 23:31:30 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=5, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=72) DEBUG 10-30 23:31:30 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=72) DEBUG 10-30 23:31:33 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) INFO: 10.134.0.45:36798 - "POST /v1/completions HTTP/1.1" 200 OK\n(APIServer pid=1) INFO 10-30 23:31:34 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.5 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 0 reqs, Waiting: [e2e-llm-inference-service] 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:31:39 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) INFO 10-30 23:31:44 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:31:49 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:31:54 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:31:59 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:32:04 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:09 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:32:14 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:19 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:32:19 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:32:24 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:29 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:32:34 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:39 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:32:44 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:49 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:32:49 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:32:54 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:32:59 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:33:04 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:09 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:33:14 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:19 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:33:19 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:33:24 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:29 [v1/engine/async_llm.py:875] [e2e-llm-inference-service] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:33:34 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 10-30 23:33:39 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 10-30 23:33:44 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:172 ### Pod tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.086315647Z","logger":"setup","caller":"runner/runner.go:247","body":"GIE build","service.name":"llm-d-epp","commit-sha":"unknown","build-ref":"","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.086521051Z","logger":"setup","caller":"runner/runner.go:271","body":"Flags processed","service.name":"llm-d-epp","flags":{"allow-experimental-plugins":false,"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1\\nkind: EndpointPickerConfig\\nplugins:\\n- type: always-disagg-pd-decider\\n- parameters:\\n deciders:\\n prefill: always-disagg-pd-decider\\n type: disagg-profile-handler\\n- type: prefill-filter\\n- type: decode-filter\\n- type: approx-prefix-cache-producer\\n- type: inflight-load-producer\\n- type: prefix-cache-affinity-filter\\n- type: token-load-scorer\\n- type: active-request-scorer\\n- type: max-score-picker\\n- parameters:\\n scheme: https\\n type: metrics-data-source\\nschedulingProfiles:\\n- name: prefill\\n plugins:\\n - pluginRef: prefill-filter\\n - pluginRef: prefix-cache-affinity-filter\\n - pluginRef: token-load-scorer\\n - pluginRef: max-score-picker\\n- name: decode\\n plugins:\\n - pluginRef: decode-filter\\n - pluginRef: active-request-scorer\\n - pluginRef: max-score-picker\\n","disable-endpoint-subset-filter":false,"drain-timeout":30000000000,"emit-endpoint-scores":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"enable-tool-calling-telemetry":false,"endpoint-selector":"","endpoint-target-ports":{},"fairness-id-metric-label-limit":1000,"feature-gates":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"ha-lease-duration":15000000000,"ha-lease-name":"","ha-renew-deadline":10000000000,"ha-retry-period":2000000000,"health-checking":false,"metrics-cert-dir":"/var/run/kserve/tls","metrics-client-ca-file":"","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"objectives-file":"","plugin-state-staleness-threshold":300000000000,"pool-group":"inference.networking.k8s.io","pool-name":"tls-verification-test-inference-pool","pool-namespace":"e2e-test-llm-tls-resources-e4cad0c3","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tls-cipher-suites":{},"tls-min-version":"","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.086687937Z","logger":"setup.trace","caller":"tracing/telemetry.go:304","body":"init OTel trace exporter","service.name":"llm-d-epp","type":"otlp","protocol":"grpc","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.087709892Z","caller":"loader/configloader.go:79","body":"Loaded raw configuration","service.name":"llm-d-epp","config":"{Plugins: [{Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\\"deciders\\":{\\"prefill\\":\\"always-disagg-pd-decider\\"}}} {Type: prefill-filter} {Type: decode-filter} {Type: approx-prefix-cache-producer} {Type: inflight-load-producer} {Type: prefix-cache-affinity-filter} {Type: token-load-scorer} {Type: active-request-scorer} {Type: max-score-picker} {Type: metrics-data-source, Parameters: {\\"scheme\\":\\"https\\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-affinity-filter} {PluginRef: token-load-scorer} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer} {PluginRef: max-score-picker}]}]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.087748484Z","logger":"setup","caller":"runner/runner.go:888","body":"Data layer: ENABLED","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","ti [e2e-llm-inference-service] mestamp":"2026-10-30T23:28:51.088241149Z","logger":"setup","caller":"runner/runner.go:439","body":"Raw config after phase one","service.name":"llm-d-epp","config":{"apiVersion":"llm-d.ai/v1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"approx-prefix-cache-producer","parameters":null,"type":"approx-prefix-cache-producer"},{"name":"inflight-load-producer","parameters":null,"type":"inflight-load-producer"},{"name":"prefix-cache-affinity-filter","parameters":null,"type":"prefix-cache-affinity-filter"},{"name":"token-load-scorer","parameters":null,"type":"token-load-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-affinity-filter","weight":null},{"pluginRef":"token-load-scorer","weight":null},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":null},{"pluginRef":"max-score-picker","weight":null}]}]},"severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.121041574Z","caller":"approximateprefix/plugin.go:182","body":"Prefix DataProducer initialized","service.name":"llm-d-epp","config":{"autoTune":true,"blockSizeTokens":64,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250,"maxSyncedTokens":2000000},"severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.121308111Z","caller":"disagg/disagg_profile_handler.go:162","body":"No deciders.encode configured, E disaggregation disabled","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.121504052Z","logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:107","body":"Creating new UtilizationDetector","service.name":"llm-d-epp","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","stalenessPolicy":"saturated","headroom":0,"severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.12191755Z","caller":"metrics/factories.go:275","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"vllm","mapping":"Mapping{all specs enabled}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.122030433Z","caller":"metrics/factories.go:275","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"sglang","mapping":"Mapping{disabled: [lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.122129233Z","caller":"metrics/factories.go:275","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"atom","mapping":"Mapping{disabled: [lora]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.122250529Z","caller":"metrics/factories.go:275","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.122435002Z","caller":"metrics/factories.go:275","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.122525379Z","caller":"metrics/factories.go:275","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"t [e2e-llm-inference-service] okenspeed","mapping":"Mapping{disabled: [lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.122609271Z","caller":"metrics/factories.go:275","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.122784588Z","caller":"loader/configloader.go:129","body":"Instantiated all plugins and applied system defaults. Effective raw configuration","service.name":"llm-d-epp","config":"{Plugins: [{Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\\"deciders\\":{\\"prefill\\":\\"always-disagg-pd-decider\\"}}} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: approx-prefix-cache-producer, Type: approx-prefix-cache-producer} {Name: inflight-load-producer, Type: inflight-load-producer} {Name: prefix-cache-affinity-filter, Type: prefix-cache-affinity-filter} {Name: token-load-scorer, Type: token-load-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\\"scheme\\":\\"https\\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: strict-band-selection-policy, Type: strict-band-selection-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: passthrough-parser, Type: passthrough-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-affinity-filter} {PluginRef: token-load-scorer, Weight: 1.00} {PluginRef: max-score-picker} {PluginRef: utilization-detector}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 1.00} {PluginRef: max-score-picker} {PluginRef: utilization-detector}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: , CrossReplica: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}, {PluginRef: passthrough-parser}]}}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.123007818Z","caller":"datalayer/data_graph.go:132","body":"auto-created default producer","service.name":"llm-d-epp","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.12313975Z","caller":"datalayer/data_graph.go:151","body":"Warning: optional data key has no producer, plugin will use fallback","service.name":"llm-d-epp","plugin":"inflight-load-producer","dataKey":"outlen-bucket/","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.123570324Z","caller":"runner/runner.go:960","body":"loaded configuration from file/text successfully","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.123629632Z","logger":"setup","caller":"runner/runner.go:467","body":"EPP config after phase two","service.name":"llm-d-epp","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/decode-filter, utilization-detector/utilization-detector], Scorers: [active-request-scorer/active-request-scorer: 1.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filt [e2e-llm-inference-service] er/prefill-filter, prefix-cache-affinity-filter/prefix-cache-affinity-filter, utilization-detector/utilization-detector], Scorers: [token-load-scorer/token-load-scorer: 1.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0x31995f606fa0 DataConfig:{Sources:[{Plugin:0x31995ee2d8c0 Extractors:[0x31995f650180]}] Syncer: SyncInterval:0s PublishTimeout:0s} FlowControlConfig: ParserRegistry:0x31995f650940 PropagatePriority:false}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.181909995Z","logger":"setup","caller":"runner/runner.go:509","body":"Setting pprof handlers","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.18196Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/profile","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.181983601Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/symbol","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.181992426Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/trace","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.181999722Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/goroutine","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182006751Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/block","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182013247Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/mutex","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182020167Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182026717Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/cmdline","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182034712Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/heap","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182041311Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/allocs","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182047486Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/threadcreate","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182064175Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/plugins/state","severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.182071942Z","logger":"setup","caller":"runner/runner.go:530","body":"parsed config","service.name":"llm-d-epp","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/decode-filter, utilization-detector/utilization-detector], Scorers: [active-request-scorer/active-request-scorer: 1.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/prefill-filter, prefix-cache-affinity-filter/prefix-cache-aff [e2e-llm-inference-service] inity-filter, utilization-detector/utilization-detector], Scorers: [token-load-scorer/token-load-scorer: 1.000000], Picker: max-score-picker/max-score-picker}]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.182139679Z","logger":"setup","caller":"datalayer/runtime.go:109","body":"Configuring datalayer runtime","service.name":"llm-d-epp","numSources":1,"severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182156663Z","logger":"setup","caller":"datalayer/runtime.go:143","body":"Processing source","service.name":"llm-d-epp","source":"metrics-data-source","numExtractors":1,"severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.182211215Z","logger":"setup","caller":"datalayer/runtime.go:187","body":"Source configured","service.name":"llm-d-epp","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"],"severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.182253684Z","logger":"setup","caller":"datalayer/runtime.go:250","body":"Datalayer runtime configured","service.name":"llm-d-epp","pollers":1,"notifiers":0,"endpointSources":1,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.182274537Z","logger":"setup","caller":"runner/runner.go:1134","body":"Flow Control layer is disabled via the flowControl feature gate, using legacy admission control","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.182422238Z","logger":"setup","caller":"runner/runner.go:395","body":"Controller manager starting","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.182479018Z","logger":"controller-runtime.metrics","caller":"server/server.go:212","body":"Starting metrics server","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.183610706Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","body":"Updated current TLS certificate","service.name":"llm-d-epp","cert":"/var/run/kserve/tls/tls.crt","key":"/var/run/kserve/tls/tls.key","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.184135517Z","logger":"controller-runtime.metrics","caller":"server/server.go:249","body":"Serving metrics server","service.name":"llm-d-epp","bindAddress":":9090","secure":true,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.184231965Z","caller":"runnable/grpc.go:37","body":"gRPC server starting","service.name":"llm-d-epp","name":"health","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.184327741Z","caller":"runnable/grpc.go:43","body":"gRPC server listening","service.name":"llm-d-epp","name":"health","port":9003,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.184508539Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:136","body":"Starting certificate poll+watcher","service.name":"llm-d-epp","cert":"/var/run/kserve/tls/tls.crt","key":"/var/run/kserve/tls/tls.key","interval":10,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.18561013Z","logger":"ext-proc","caller":"server/runserver.go:257","body":"server TLS","service.name":"llm-d-epp","tls":true,"cert_path":"/var/run/kserve/tls","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.185684657Z","caller":"runnable/grpc.go:37","body":"gRPC server starting","service.name":"llm-d-epp","name":"ext-proc","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.18575773Z","caller":"runnable/grpc.go:43","body":"gRPC server listening","service.name":"llm-d-epp","name":"ext-proc","port":9002,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.190740914Z","caller":"controller/controller.go:372","body":"Starting EventSource","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io", [e2e-llm-inference-service] "controllerKind":"InferencePool","source":"kind source: *v1.InferencePool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.19194537Z","caller":"controller/controller.go:372","body":"Starting EventSource","service.name":"llm-d-epp","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.192456129Z","caller":"controller/controller.go:372","body":"Starting EventSource","service.name":"llm-d-epp","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1.InferenceObjective","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.192551737Z","caller":"controller/controller.go:372","body":"Starting EventSource","service.name":"llm-d-epp","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1.InferencePool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.192816633Z","caller":"controller/controller.go:372","body":"Starting EventSource","service.name":"llm-d-epp","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.193257771Z","caller":"controller/controller.go:372","body":"Starting EventSource","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod","severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.198313368Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.37.1/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.198645959Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.37.1/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.199385429Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.37.1/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.200456116Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.37.1/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.204847088Z","logger":"controller-runtime.cache","caller":"rest/warnings.go:107","body":"Warning: llm-d.ai/v1alpha2 InferenceObjective is deprecated; use llm-d.ai/v1","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.205085541Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.37.1/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.396846095Z","caller":"controller/controller.go:305","body":"Starting Controller","service.name":"llm-d-epp","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.396876695Z","caller":"controller/controller.go:308","body":"Starting workers","service.name":"llm-d-epp","controlle [e2e-llm-inference-service] r":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.396785285Z","caller":"controller/controller.go:305","body":"Starting Controller","service.name":"llm-d-epp","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.396913733Z","caller":"controller/controller.go:308","body":"Starting workers","service.name":"llm-d-epp","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.396911597Z","caller":"controller/controller.go:305","body":"Starting Controller","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.39694195Z","caller":"controller/controller.go:308","body":"Starting workers","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1,"severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:51.397151197Z","caller":"controller/inferencepool_reconciler.go:49","body":"Reconciling InferencePool","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"tls-verification-test-inference-pool","namespace":"e2e-test-llm-tls-resources-e4cad0c3"},"namespace":"e2e-test-llm-tls-resources-e4cad0c3","name":"tls-verification-test-inference-pool","reconcileID":"60ea3cf5-71de-4494-bd62-06bcc5964997","severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.397739111Z","caller":"controller/controller.go:305","body":"Starting Controller","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:28:51.397798259Z","caller":"controller/controller.go:308","body":"Starting workers","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1,"severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:28:58.011796064Z","caller":"controller/inferencepool_reconciler.go:49","body":"Reconciling InferencePool","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"tls-verification-test-inference-pool","namespace":"e2e-test-llm-tls-resources-e4cad0c3"},"namespace":"e2e-test-llm-tls-resources-e4cad0c3","name":"tls-verification-test-inference-pool","reconcileID":"4b874d1f-5008-416f-af4a-063b27543f5d","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:29:23.421177727Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:241","body":"certificate event","service.name":"llm-d-epp","cert":"/var/run/kserve/tls/tls.crt","key":"/var/run/kserve/tls/tls.key","event":"CHMOD \\"/var/run/kserve/tls/tls.key\\"","severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-10-30T23:29:23.421843021Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:214","body":"Updated current TLS certificate","service.name":"llm-d-epp","cert":"/var/run/kserve/tls/tls.crt","key":"/var/run/kserve/tls/tls.key","severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:29:23.421880896Z","logger":"controller-runtime.certwatcher","caller":"certwatcher/certwatcher.go:241","body":"certificate event","service.name":"llm-d-epp","cert":"/var/run/kserve/tls/tls.crt","key":"/var/run/kserve/tls/tls.key","event":"CHMOD \\"/var/run/kserve/tls/tls.crt\\"","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10 [e2e-llm-inference-service] -30T23:30:49.350200066Z","caller":"inflightload/producer.go:379","body":"Injected dynamic attribute into endpoint","service.name":"llm-d-epp","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-656f77bc69-6b9c8-rank-0","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:30:49.3502636Z","caller":"datastore/datastore.go:407","body":"Pod added","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"tls-verification-test-kserve-656f77bc69-6b9c8","namespace":"e2e-test-llm-tls-resources-e4cad0c3"},"namespace":"e2e-test-llm-tls-resources-e4cad0c3","name":"tls-verification-test-kserve-656f77bc69-6b9c8","reconcileID":"c02f1a9a-006d-42e6-8ab9-1faae60215b1","name":"tls-verification-test-kserve-656f77bc69-6b9c8","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:25.022709613Z","caller":"handlers/server.go:582","body":"EPP received request","service.name":"llm-d-epp","x-request-id":"5316dc5b-24ed-4e99-a43f-7c16a735c9aa","eppRequestID":"1908bdae-8330-4417-9169-6c7ccc7e2fa2","trace_id":"abc403908265290a981aa51a2e2e8351","span_id":"17c9952ad05263da","severity_number":5}\n{"severity_text":"ERROR","timestamp":"2026-10-30T23:31:25.023069528Z","caller":"handlers/server.go:784","body":"Failed to process request","service.name":"llm-d-epp","x-request-id":"5316dc5b-24ed-4e99-a43f-7c16a735c9aa","eppRequestID":"1908bdae-8330-4417-9169-6c7ccc7e2fa2","trace_id":"abc403908265290a981aa51a2e2e8351","span_id":"17c9952ad05263da","error":"inference error: ServiceUnavailable - no endpoints available for the given request","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/pkg/epp/handlers.(*StreamingServer).Process\\n\\t/workspace/pkg/epp/handlers/server.go:784\\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\\n\\t/opt/app-root/src/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\\ngoogle.golang.org/grpc.(*Server).processRPC\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1450\\ngoogle.golang.org/grpc.(*Server).handleStream\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1595\\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1096"}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:25.038787878Z","caller":"handlers/server.go:582","body":"EPP received request","service.name":"llm-d-epp","x-request-id":"617f7978-32be-44fb-ab03-8df58b207acb","eppRequestID":"6a88dd3d-4d31-49f1-a0ea-5c363417e938","trace_id":"6e764534ea0bdf708fd4fd9ac368920e","span_id":"258d3087c5f97f65","severity_number":5}\n{"severity_text":"ERROR","timestamp":"2026-10-30T23:31:25.038989716Z","caller":"handlers/server.go:784","body":"Failed to process request","service.name":"llm-d-epp","x-request-id":"617f7978-32be-44fb-ab03-8df58b207acb","eppRequestID":"6a88dd3d-4d31-49f1-a0ea-5c363417e938","trace_id":"6e764534ea0bdf708fd4fd9ac368920e","span_id":"258d3087c5f97f65","error":"inference error: ServiceUnavailable - no endpoints available for the given request","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/pkg/epp/handlers.(*StreamingServer).Process\\n\\t/workspace/pkg/epp/handlers/server.go:784\\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\\n\\t/opt/app-root/src/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\\ngoogle.golang.org/grpc.(*Server).processRPC\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1450\\ngoogle.golang.org/grpc.(*Server).handleStream\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1595\\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1096"}\n{"severity_text" [e2e-llm-inference-service] :"DEBUG","timestamp":"2026-10-30T23:31:29.053904854Z","caller":"handlers/server.go:582","body":"EPP received request","service.name":"llm-d-epp","x-request-id":"12cf40c6-3b5a-4de1-8476-389b929a3faa","eppRequestID":"886a7de5-38fc-4e09-b1c3-68545612eaea","trace_id":"20e3a3535d2b720191cad961112eb385","span_id":"b77e3f16f3b5f4d4","severity_number":5}\n{"severity_text":"ERROR","timestamp":"2026-10-30T23:31:29.054183154Z","caller":"handlers/server.go:784","body":"Failed to process request","service.name":"llm-d-epp","x-request-id":"12cf40c6-3b5a-4de1-8476-389b929a3faa","eppRequestID":"886a7de5-38fc-4e09-b1c3-68545612eaea","trace_id":"20e3a3535d2b720191cad961112eb385","span_id":"b77e3f16f3b5f4d4","error":"inference error: ServiceUnavailable - no endpoints available for the given request","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/pkg/epp/handlers.(*StreamingServer).Process\\n\\t/workspace/pkg/epp/handlers/server.go:784\\ngithub.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler\\n\\t/opt/app-root/src/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.37.0/service/ext_proc/v3/external_processor_grpc.pb.go:106\\ngoogle.golang.org/grpc.(*Server).processRPC\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1450\\ngoogle.golang.org/grpc.(*Server).handleStream\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1595\\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\\n\\t/opt/app-root/src/go/pkg/mod/google.golang.org/grpc@v1.84.0/server.go:1096"}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:29.540290693Z","caller":"inflightload/producer.go:379","body":"Injected dynamic attribute into endpoint","service.name":"llm-d-epp","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq-rank-0","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:29.540354719Z","caller":"datastore/datastore.go:407","body":"Pod added","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"tls-verification-test-kserve-prefill-577b5bd485-cjmrq","namespace":"e2e-test-llm-tls-resources-e4cad0c3"},"namespace":"e2e-test-llm-tls-resources-e4cad0c3","name":"tls-verification-test-kserve-prefill-577b5bd485-cjmrq","reconcileID":"4def9d1d-40e6-4065-a6ea-003ba488883c","name":"tls-verification-test-kserve-prefill-577b5bd485-cjmrq","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:30.360787387Z","caller":"handlers/server.go:582","body":"EPP received request","service.name":"llm-d-epp","x-request-id":"edc30809-3e8d-45f9-b900-092a861cf69e","eppRequestID":"fc9028bc-5f2c-4e16-8e05-d88e7fda8fd2","trace_id":"6bdafb0038e44f5eeb9266cfce025adb","span_id":"58ccfd41e994f6a1","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:30.361360908Z","caller":"handlers/server.go:984","body":"EPP sent request body response(s) to proxy","service.name":"llm-d-epp","x-request-id":"edc30809-3e8d-45f9-b900-092a861cf69e","eppRequestID":"fc9028bc-5f2c-4e16-8e05-d88e7fda8fd2","trace_id":"6bdafb0038e44f5eeb9266cfce025adb","span_id":"58ccfd41e994f6a1","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:36.611467829Z","caller":"handlers/server.go:1012","body":"EPP sent response body back to proxy","service.name":"llm-d-epp","x-request-id":"edc30809-3e8d-45f9-b900-092a861cf69e","eppRequestID":"fc9028bc-5f2c-4e16-8e05-d88e7fda8fd2","trace_id":"6bdafb0038e44f5eeb9266cfce025adb","span_id":"58ccfd41e994f6a1","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:37.069380285Z","caller":"handlers/server.go:582","body":"EPP received request","service.name":"llm-d-epp","x-request-id":"bdd0dd46-6cc0-4eeb-a442-657ef7861f5e","eppRequestID":"b003cd2c-c180-49cf-ae62-d6834689ebb7","trace_id":"65983e9936639bca71429b2c3a5766dc","span_id":"1bfdb80fc1b249cf","s [e2e-llm-inference-service] everity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:37.069816225Z","caller":"handlers/server.go:984","body":"EPP sent request body response(s) to proxy","service.name":"llm-d-epp","x-request-id":"bdd0dd46-6cc0-4eeb-a442-657ef7861f5e","eppRequestID":"b003cd2c-c180-49cf-ae62-d6834689ebb7","trace_id":"65983e9936639bca71429b2c3a5766dc","span_id":"1bfdb80fc1b249cf","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-10-30T23:31:40.459861948Z","caller":"handlers/server.go:1012","body":"EPP sent response body back to proxy","service.name":"llm-d-epp","x-request-id":"bdd0dd46-6cc0-4eeb-a442-657ef7861f5e","eppRequestID":"b003cd2c-c180-49cf-ae62-d6834689ebb7","trace_id":"65983e9936639bca71429b2c3a5766dc","span_id":"1bfdb80fc1b249cf","severity_number":5}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 49427752-51b5-4566-84c9-f35aac86b87c [e2e-llm-inference-service] resourceVersion: '66881' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.83 [e2e-llm-inference-service] nodeName: ip-10-0-128-97.us-east-2.compute.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5 [e2e-llm-inference-service] uid: 86b1ad02-e855-4430-abd6-9435d5a0afa7 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: ca6097b6-519f-4301-9bab-849bf24910c3 [e2e-llm-inference-service] resourceVersion: '67929' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: placeholder [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.90 [e2e-llm-inference-service] nodeName: ip-10-0-138-251.us-east-2.compute.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-577b5bd485-cjmrq [e2e-llm-inference-service] uid: 9c6f7b92-edd2-4257-98ce-1b0f032f15e6 [e2e-llm-inference-service] - ip: 10.134.0.45 [e2e-llm-inference-service] nodeName: ip-10-0-132-192.us-east-2.compute.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-656f77bc69-6b9c8 [e2e-llm-inference-service] uid: 053dde5c-1ffd-47c4-8a1c-85c7187dc621 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-656f77bc69-6b9c8 [e2e-llm-inference-service] generateName: tls-verification-test-kserve-656f77bc69- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 053dde5c-1ffd-47c4-8a1c-85c7187dc621 [e2e-llm-inference-service] resourceVersion: '67579' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 656f77bc69 [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] topology.kubernetes.io/region: us-east-2 [e2e-llm-inference-service] topology.kubernetes.io/zone: us-east-2a [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.45/23"],"mac_address":"0a:58:0a:86:00:2d","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.45/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.45\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:2d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: tls-verification-test-kserve-656f77bc69 [e2e-llm-inference-service] uid: 19dca3ec-bfc0-4951-858d-6e34578e45a4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-192 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"19dca3ec-bfc0-4951-858d-6e34578e45a4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"KSERVE_TRACING_CONFIG_VERSION"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.45"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-wck6v [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:odh-v3.6.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: KSERVE_TRACING_CONFIG_VERSION [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: tls-verification-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-wck6v [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001080000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] - name: kube-api-access-wck6v [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001080000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n if\ [e2e-llm-inference-service] \ [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\")\ [e2e-llm-inference-service] \ | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n \ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n fi\n\ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n \ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep -H\ [e2e-llm-inference-service] \ . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills\ [e2e-llm-inference-service] \ the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # KSERVE_KV_TRANSFER_ARGS\ [e2e-llm-inference-service] \ is set as a container env var by the config template;\n # it is empty (no-op)\ [e2e-llm-inference-service] \ when KV cache offloading is not configured.\n if [ -n \"${KSERVE_KV_TRANSFER_ARGS:-}\"\ [e2e-llm-inference-service] \ ]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\n if [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\\ [e2e-llm-inference-service] .[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.22.0\" ]; then\n KV_TRANSFER_ARGS=\"${KSERVE_KV_TRANSFER_ARGS}\"\ [e2e-llm-inference-service] \n fi\n fi\n # This template is only composed for a disaggregated P/D topology\ [e2e-llm-inference-service] \ (spec.prefill set).\n # Decode is the KV consumer; without a connector here\ [e2e-llm-inference-service] \ it recomputes prefill's KV.\n if [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n \ [e2e-llm-inference-service] \ # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n\ [e2e-llm-inference-service] \ # raise RuntimeError(\"NIXL is not available\") and never finish starting.\n\ [e2e-llm-inference-service] \ NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n\ [e2e-llm-inference-service] \ \"${NIXL_PY}\" ] && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n\ [e2e-llm-inference-service] \ KV_TRANSFER_ARGS=\"--kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\ [e2e-llm-inference-service] \",\\\"kv_role\\\":\\\"kv_consumer\\\"}'\"\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ available, enabling NixlConnector (kv_consumer)\"\n else\n echo \"\ [e2e-llm-inference-service] [kv-transfer] NIXL not available, P/D KV transfer stays disabled\"\n fi\n\ [e2e-llm-inference-service] \ fi\nfi\n\neval \"exec vllm serve ${KSERVE_MODEL_ARGS:-/mnt/models} \\\n \ [e2e-llm-inference-service] \ --served-model-name \"facebook/opt-125m\" \"publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-wck6v [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001080000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-kserve [e2e-llm-inference-service] serviceAccount: tls-verification-test-kserve [e2e-llm-inference-service] nodeName: ip-10-0-132-192.us-east-2.compute.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c33,c12 [e2e-llm-inference-service] fsGroup: 1001080000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjgbt [e2e-llm-inference-service] - name: tls-verification-test-kserve-dockercfg-h85jx [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:53Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] hostIP: 10.0.132.192 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.192 [e2e-llm-inference-service] podIP: 10.134.0.45 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.45 [e2e-llm-inference-service] startTime: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:odh-v3.6.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:b7fdc33c56faa4ff6806a0ae6e1724e0a40260a4cfa97ea23e1743be0755f0b5 [e2e-llm-inference-service] containerID: cri-o://0b92c118c2e5c6a75763a8349f6b2efddba35fb0805d2533c5f7349cf41b40d6 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-wck6v [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001080000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001080000 [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] finishedAt: '2026-10-30T23:28:52Z' [e2e-llm-inference-service] containerID: cri-o://2e91ed9f4504930bd8b725646fee1586394c1b20bc5e8f1f7a5cbb4e69bd1259 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] containerID: cri-o://2e91ed9f4504930bd8b725646fee1586394c1b20bc5e8f1f7a5cbb4e69bd1259 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-wck6v [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001080000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001080000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-10-30T23:28:53Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://58528918074e987e9cfb7a99cf37e777cbc4a8130ed6fe683d89b67ed00f7d3a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-wck6v [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001080000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001080000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-577b5bd485-cjmrq [e2e-llm-inference-service] generateName: tls-verification-test-kserve-prefill-577b5bd485- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 9c6f7b92-edd2-4257-98ce-1b0f032f15e6 [e2e-llm-inference-service] resourceVersion: '67928' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 577b5bd485 [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] topology.kubernetes.io/region: us-east-2 [e2e-llm-inference-service] topology.kubernetes.io/zone: us-east-2a [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.90/23"],"mac_address":"0a:58:0a:84:00:5a","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.90/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.90\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:5a\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-577b5bd485 [e2e-llm-inference-service] uid: fb489c4e-fe3f-4009-a217-3c2a7211d21c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-138-251 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"fb489c4e-fe3f-4009-a217-3c2a7211d21c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.90"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-tw9rw [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] - name: kube-api-access-tw9rw [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001080000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n if\ [e2e-llm-inference-service] \ [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\")\ [e2e-llm-inference-service] \ | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n \ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n fi\n\ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n \ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep -H\ [e2e-llm-inference-service] \ . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills\ [e2e-llm-inference-service] \ the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # KSERVE_KV_TRANSFER_ARGS\ [e2e-llm-inference-service] \ is set as a container env var by the config template;\n # it is empty (no-op)\ [e2e-llm-inference-service] \ when KV cache offloading is not configured.\n if [ -n \"${KSERVE_KV_TRANSFER_ARGS:-}\"\ [e2e-llm-inference-service] \ ]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\n if [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\\ [e2e-llm-inference-service] .[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.22.0\" ]; then\n KV_TRANSFER_ARGS=\"${KSERVE_KV_TRANSFER_ARGS}\"\ [e2e-llm-inference-service] \n fi\n fi\n # This template is only composed for a disaggregated P/D topology\ [e2e-llm-inference-service] \ (spec.prefill set).\n # Prefill is the KV producer; without a connector here\ [e2e-llm-inference-service] \ decode has nothing to fetch.\n if [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n\ [e2e-llm-inference-service] \ # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n\ [e2e-llm-inference-service] \ # raise RuntimeError(\"NIXL is not available\") and never finish starting.\n\ [e2e-llm-inference-service] \ NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n\ [e2e-llm-inference-service] \ \"${NIXL_PY}\" ] && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n\ [e2e-llm-inference-service] \ KV_TRANSFER_ARGS=\"--kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\ [e2e-llm-inference-service] \",\\\"kv_role\\\":\\\"kv_producer\\\"}'\"\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ available, enabling NixlConnector (kv_producer)\"\n else\n echo \"\ [e2e-llm-inference-service] [kv-transfer] NIXL not available, P/D KV transfer stays disabled\"\n fi\n\ [e2e-llm-inference-service] \ fi\nfi\n\neval \"exec vllm serve ${KSERVE_MODEL_ARGS:-/mnt/models} \\\n \ [e2e-llm-inference-service] \ --served-model-name \"facebook/opt-125m\" \\\n --port 8000 \\\n --root-path\ [e2e-llm-inference-service] \ /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS}\ [e2e-llm-inference-service] \ \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-tw9rw [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001080000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-138-251.us-east-2.compute.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c33,c12 [e2e-llm-inference-service] fsGroup: 1001080000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjgbt [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:53Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] hostIP: 10.0.138.251 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.138.251 [e2e-llm-inference-service] podIP: 10.132.0.90 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.90 [e2e-llm-inference-service] startTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] finishedAt: '2026-10-30T23:28:52Z' [e2e-llm-inference-service] containerID: cri-o://65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] containerID: cri-o://65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-tw9rw [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001080000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001080000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-10-30T23:28:53Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-tw9rw [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001080000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001080000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5 [e2e-llm-inference-service] generateName: tls-verification-test-kserve-router-scheduler-69f7c679f8- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 86b1ad02-e855-4430-abd6-9435d5a0afa7 [e2e-llm-inference-service] resourceVersion: '66880' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69f7c679f8 [e2e-llm-inference-service] topology.kubernetes.io/region: us-east-2 [e2e-llm-inference-service] topology.kubernetes.io/zone: us-east-2a [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 1.0.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.83/23"],"mac_address":"0a:58:0a:85:00:53","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.83/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.83\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:53\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-69f7c679f8 [e2e-llm-inference-service] uid: a08fcff0-c051-4bde-91bf-c8789cfd40ad [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-97 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a08fcff0-c051-4bde-91bf-c8789cfd40ad"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.83"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-4rd9h [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:odh-v3.6.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - tls-verification-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --metrics-cert-dir=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1\nkind: EndpointPickerConfig\nplugins:\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ approx-prefix-cache-producer\n- type: inflight-load-producer\n- type: prefix-cache-affinity-filter\n\ [e2e-llm-inference-service] - type: token-load-scorer\n- type: active-request-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-affinity-filter\n\ [e2e-llm-inference-service] \ - pluginRef: token-load-scorer\n - pluginRef: max-score-picker\n- name:\ [e2e-llm-inference-service] \ decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-4rd9h [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001080000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-epp-sa [e2e-llm-inference-service] serviceAccount: tls-verification-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-128-97.us-east-2.compute.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c33,c12 [e2e-llm-inference-service] fsGroup: 1001080000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjgbt [e2e-llm-inference-service] - name: tls-verification-test-epp-sa-dockercfg-67cr4 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:51Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] hostIP: 10.0.128.97 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.97 [e2e-llm-inference-service] podIP: 10.133.0.83 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.83 [e2e-llm-inference-service] startTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:odh-v3.6.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:1180e3a984930af30a4a71f0f819c79fa358976693f0de355dc65bd5da2b3d3a [e2e-llm-inference-service] containerID: cri-o://1b2e0615c1225ad2659d76181f52f5433f84a217a9dcbe3eefb7ef0ce161baa1 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-4rd9h [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001080000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001080000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 1f2c6eb9-bca7-4754-a7f9-49fc35800fcf [e2e-llm-inference-service] resourceVersion: '66258' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: tls-verification-test-epp-sa-dockercfg-67cr4 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"tls-verification-test-epp-sa-dockercfg-67cr4"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjgbt"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-fjgbt [e2e-llm-inference-service] - name: tls-verification-test-epp-sa-dockercfg-67cr4 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjgbt [e2e-llm-inference-service] - name: tls-verification-test-epp-sa-dockercfg-67cr4 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 3aa21325-c68a-4dc2-9a2c-a2408251d944 [e2e-llm-inference-service] resourceVersion: '65848' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: tls-verification-test-kserve-dockercfg-h85jx [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"tls-verification-test-kserve-dockercfg-h85jx"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fjgbt"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-fjgbt [e2e-llm-inference-service] - name: tls-verification-test-kserve-dockercfg-h85jx [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fjgbt [e2e-llm-inference-service] - name: tls-verification-test-kserve-dockercfg-h85jx [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 47c14a30-0396-421b-96c6-33b83b46b840 [e2e-llm-inference-service] resourceVersion: '66284' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.30.25 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.30.25 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 6328d582-cfc9-426a-8fd4-4dbff2b77d8d [e2e-llm-inference-service] resourceVersion: '66247' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: placeholder [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.237.186 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.237.186 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 722fe9e1-da7d-4889-95b4-158f2934cf42 [e2e-llm-inference-service] resourceVersion: '67585' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"KSERVE_TRACING_CONFIG_VERSION"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:terminatingReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:odh-v3.6.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: KSERVE_TRACING_CONFIG_VERSION [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: tls-verification-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n \ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\"\ [e2e-llm-inference-service] ) | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n\ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n \ [e2e-llm-inference-service] \ fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n\ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep\ [e2e-llm-inference-service] \ -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe\ [e2e-llm-inference-service] \ only fills the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"\ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # KSERVE_KV_TRANSFER_ARGS\ [e2e-llm-inference-service] \ is set as a container env var by the config template;\n # it is empty\ [e2e-llm-inference-service] \ (no-op) when KV cache offloading is not configured.\n if [ -n \"${KSERVE_KV_TRANSFER_ARGS:-}\"\ [e2e-llm-inference-service] \ ]; then\n # --kv-transfer-config with OffloadingConnector requires\ [e2e-llm-inference-service] \ vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ \"$VLLM_VERSION\" =~\ [e2e-llm-inference-service] \ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] ${KSERVE_KV_TRANSFER_ARGS}\"\n fi\n fi\n # This template is only composed\ [e2e-llm-inference-service] \ for a disaggregated P/D topology (spec.prefill set).\n # Decode is the\ [e2e-llm-inference-service] \ KV consumer; without a connector here it recomputes prefill's KV.\n if\ [e2e-llm-inference-service] \ [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject when NIXL is importable.\ [e2e-llm-inference-service] \ CPU and other non-NIXL engine images\n # raise RuntimeError(\"NIXL\ [e2e-llm-inference-service] \ is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ]\ [e2e-llm-inference-service] \ && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_consumer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_consumer)\"\n else\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ not available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\n\ [e2e-llm-inference-service] eval \"exec vllm serve ${KSERVE_MODEL_ARGS:-/mnt/models} \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-kserve [e2e-llm-inference-service] serviceAccount: tls-verification-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] terminatingReplicas: 0 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "tls-verification-test-kserve-656f77bc69" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: b464b2e3-7446-43b9-bc2e-797e9f98baab [e2e-llm-inference-service] resourceVersion: '67934' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:terminatingReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n \ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\"\ [e2e-llm-inference-service] ) | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n\ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n \ [e2e-llm-inference-service] \ fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n\ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep\ [e2e-llm-inference-service] \ -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe\ [e2e-llm-inference-service] \ only fills the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"\ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # KSERVE_KV_TRANSFER_ARGS\ [e2e-llm-inference-service] \ is set as a container env var by the config template;\n # it is empty\ [e2e-llm-inference-service] \ (no-op) when KV cache offloading is not configured.\n if [ -n \"${KSERVE_KV_TRANSFER_ARGS:-}\"\ [e2e-llm-inference-service] \ ]; then\n # --kv-transfer-config with OffloadingConnector requires\ [e2e-llm-inference-service] \ vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ \"$VLLM_VERSION\" =~\ [e2e-llm-inference-service] \ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] ${KSERVE_KV_TRANSFER_ARGS}\"\n fi\n fi\n # This template is only composed\ [e2e-llm-inference-service] \ for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the\ [e2e-llm-inference-service] \ KV producer; without a connector here decode has nothing to fetch.\n \ [e2e-llm-inference-service] \ if [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject when NIXL is\ [e2e-llm-inference-service] \ importable. CPU and other non-NIXL engine images\n # raise RuntimeError(\"\ [e2e-llm-inference-service] NIXL is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ]\ [e2e-llm-inference-service] \ && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_producer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_producer)\"\n else\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ not available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\n\ [e2e-llm-inference-service] eval \"exec vllm serve ${KSERVE_MODEL_ARGS:-/mnt/models} \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] terminatingReplicas: 0 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "tls-verification-test-kserve-prefill-577b5bd485" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 080e49da-b07b-43f3-91b9-4c00d677531c [e2e-llm-inference-service] resourceVersion: '66884' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:terminatingReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 1.0.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:odh-v3.6.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - tls-verification-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --metrics-cert-dir=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\n- type: prefill-filter\n\ [e2e-llm-inference-service] - type: decode-filter\n- type: approx-prefix-cache-producer\n- type: inflight-load-producer\n\ [e2e-llm-inference-service] - type: prefix-cache-affinity-filter\n- type: token-load-scorer\n- type:\ [e2e-llm-inference-service] \ active-request-scorer\n- type: max-score-picker\n- parameters:\n scheme:\ [e2e-llm-inference-service] \ https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n\ [e2e-llm-inference-service] \ plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-affinity-filter\n\ [e2e-llm-inference-service] \ - pluginRef: token-load-scorer\n - pluginRef: max-score-picker\n- name:\ [e2e-llm-inference-service] \ decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-epp-sa [e2e-llm-inference-service] serviceAccount: tls-verification-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] terminatingReplicas: 0 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] lastTransitionTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "tls-verification-test-kserve-router-scheduler-69f7c679f8" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-656f77bc69 [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 19dca3ec-bfc0-4951-858d-6e34578e45a4 [e2e-llm-inference-service] resourceVersion: '67584' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 656f77bc69 [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] uid: 722fe9e1-da7d-4889-95b4-158f2934cf42 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"722fe9e1-da7d-4889-95b4-158f2934cf42"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"KSERVE_TRACING_CONFIG_VERSION"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:30:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:terminatingReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 656f77bc69 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 656f77bc69 [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:odh-v3.6.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: KSERVE_TRACING_CONFIG_VERSION [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: tls-verification-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n \ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\"\ [e2e-llm-inference-service] ) | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n\ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n \ [e2e-llm-inference-service] \ fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n\ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep\ [e2e-llm-inference-service] \ -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe\ [e2e-llm-inference-service] \ only fills the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"\ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # KSERVE_KV_TRANSFER_ARGS\ [e2e-llm-inference-service] \ is set as a container env var by the config template;\n # it is empty\ [e2e-llm-inference-service] \ (no-op) when KV cache offloading is not configured.\n if [ -n \"${KSERVE_KV_TRANSFER_ARGS:-}\"\ [e2e-llm-inference-service] \ ]; then\n # --kv-transfer-config with OffloadingConnector requires\ [e2e-llm-inference-service] \ vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ \"$VLLM_VERSION\" =~\ [e2e-llm-inference-service] \ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] ${KSERVE_KV_TRANSFER_ARGS}\"\n fi\n fi\n # This template is only composed\ [e2e-llm-inference-service] \ for a disaggregated P/D topology (spec.prefill set).\n # Decode is the\ [e2e-llm-inference-service] \ KV consumer; without a connector here it recomputes prefill's KV.\n if\ [e2e-llm-inference-service] \ [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject when NIXL is importable.\ [e2e-llm-inference-service] \ CPU and other non-NIXL engine images\n # raise RuntimeError(\"NIXL\ [e2e-llm-inference-service] \ is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ]\ [e2e-llm-inference-service] \ && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_consumer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_consumer)\"\n else\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ not available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\n\ [e2e-llm-inference-service] eval \"exec vllm serve ${KSERVE_MODEL_ARGS:-/mnt/models} \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-kserve [e2e-llm-inference-service] serviceAccount: tls-verification-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] terminatingReplicas: 0 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-577b5bd485 [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: fb489c4e-fe3f-4009-a217-3c2a7211d21c [e2e-llm-inference-service] resourceVersion: '67933' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 577b5bd485 [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill [e2e-llm-inference-service] uid: b464b2e3-7446-43b9-bc2e-797e9f98baab [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b464b2e3-7446-43b9-bc2e-797e9f98baab"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:terminatingReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 577b5bd485 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 577b5bd485 [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:5d3a44346df74740a90b1bbd421ba27d3a9369e12753aec4f5f20b696767b937 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n \ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\"\ [e2e-llm-inference-service] ) | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n\ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n \ [e2e-llm-inference-service] \ fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n\ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep\ [e2e-llm-inference-service] \ -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe\ [e2e-llm-inference-service] \ only fills the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"\ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # KSERVE_KV_TRANSFER_ARGS\ [e2e-llm-inference-service] \ is set as a container env var by the config template;\n # it is empty\ [e2e-llm-inference-service] \ (no-op) when KV cache offloading is not configured.\n if [ -n \"${KSERVE_KV_TRANSFER_ARGS:-}\"\ [e2e-llm-inference-service] \ ]; then\n # --kv-transfer-config with OffloadingConnector requires\ [e2e-llm-inference-service] \ vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ \"$VLLM_VERSION\" =~\ [e2e-llm-inference-service] \ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] ${KSERVE_KV_TRANSFER_ARGS}\"\n fi\n fi\n # This template is only composed\ [e2e-llm-inference-service] \ for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the\ [e2e-llm-inference-service] \ KV producer; without a connector here decode has nothing to fetch.\n \ [e2e-llm-inference-service] \ if [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject when NIXL is\ [e2e-llm-inference-service] \ importable. CPU and other non-NIXL engine images\n # raise RuntimeError(\"\ [e2e-llm-inference-service] NIXL is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ]\ [e2e-llm-inference-service] \ && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_producer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_producer)\"\n else\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ not available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\n\ [e2e-llm-inference-service] eval \"exec vllm serve ${KSERVE_MODEL_ARGS:-/mnt/models} \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] terminatingReplicas: 0 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-69f7c679f8 [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: a08fcff0-c051-4bde-91bf-c8789cfd40ad [e2e-llm-inference-service] resourceVersion: '66883' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69f7c679f8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 080e49da-b07b-43f3-91b9-4c00d677531c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"080e49da-b07b-43f3-91b9-4c00d677531c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:terminatingReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69f7c679f8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69f7c679f8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 1.0.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:odh-v3.6.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - tls-verification-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --metrics-cert-dir=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\n- type: prefill-filter\n\ [e2e-llm-inference-service] - type: decode-filter\n- type: approx-prefix-cache-producer\n- type: inflight-load-producer\n\ [e2e-llm-inference-service] - type: prefix-cache-affinity-filter\n- type: token-load-scorer\n- type:\ [e2e-llm-inference-service] \ active-request-scorer\n- type: max-score-picker\n- parameters:\n scheme:\ [e2e-llm-inference-service] \ https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n\ [e2e-llm-inference-service] \ plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-affinity-filter\n\ [e2e-llm-inference-service] \ - pluginRef: token-load-scorer\n - pluginRef: max-score-picker\n- name:\ [e2e-llm-inference-service] \ decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-epp-sa [e2e-llm-inference-service] serviceAccount: tls-verification-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] terminatingReplicas: 0 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: d9cfd0aa-863a-4e38-aa93-dbae0d7f82f2 [e2e-llm-inference-service] resourceVersion: '66267' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: tls-verification-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: tls-verification-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 65e2eeb4-0bd2-498c-acfe-90873a054f6d [e2e-llm-inference-service] resourceVersion: '65857' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: tls-verification-test-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 6b94064f-298e-4c70-8068-f1a7c1fcc170 [e2e-llm-inference-service] resourceVersion: '66265' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - patch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - events [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 625ac0ec-8963-4fb8-8d54-7ea2d401d939 [e2e-llm-inference-service] resourceVersion: '65852' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-service-6hv9p [e2e-llm-inference-service] generateName: tls-verification-test-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 886ab02d-95c8-4367-bb21-a3648bbcb477 [e2e-llm-inference-service] resourceVersion: '66882' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: tls-verification-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] uid: 47c14a30-0396-421b-96c6-33b83b46b840 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:29:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47c14a30-0396-421b-96c6-33b83b46b840"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.83 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5 [e2e-llm-inference-service] uid: 86b1ad02-e855-4430-abd6-9435d5a0afa7 [e2e-llm-inference-service] nodeName: ip-10-0-128-97.us-east-2.compute.internal [e2e-llm-inference-service] zone: us-east-2a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc-5nhfn [e2e-llm-inference-service] generateName: tls-verification-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 9b5c882d-e6f2-4764-ad3b-b3eb7746c222 [e2e-llm-inference-service] resourceVersion: '67931' [e2e-llm-inference-service] generation: 5 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: placeholder [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] uid: 6328d582-cfc9-426a-8fd4-4dbff2b77d8d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:31:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:serving.kserve.io/llmisvc-revision: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6328d582-cfc9-426a-8fd4-4dbff2b77d8d"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.45 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-656f77bc69-6b9c8 [e2e-llm-inference-service] uid: 053dde5c-1ffd-47c4-8a1c-85c7187dc621 [e2e-llm-inference-service] nodeName: ip-10-0-132-192.us-east-2.compute.internal [e2e-llm-inference-service] zone: us-east-2a [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.90 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-577b5bd485-cjmrq [e2e-llm-inference-service] uid: 9c6f7b92-edd2-4257-98ce-1b0f032f15e6 [e2e-llm-inference-service] nodeName: ip-10-0-138-251.us-east-2.compute.internal [e2e-llm-inference-service] zone: us-east-2a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: d9cfd0aa-863a-4e38-aa93-dbae0d7f82f2 [e2e-llm-inference-service] resourceVersion: '66267' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-tls-resources-e4cad0c3:tls-verification-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 65e2eeb4-0bd2-498c-acfe-90873a054f6d [e2e-llm-inference-service] resourceVersion: '65857' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-tls-resources-e4cad0c3:tls-verification-test-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 6b94064f-298e-4c70-8068-f1a7c1fcc170 [e2e-llm-inference-service] resourceVersion: '66265' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - patch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - events [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 625ac0ec-8963-4fb8-8d54-7ea2d401d939 [e2e-llm-inference-service] resourceVersion: '65852' [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-10-30T23:28:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parentRefs: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66549' [e2e-llm-inference-service] uid: cec4ef32-b856-4a25-bb76-f64c765c15fb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/completions [e2e-llm-inference-service] name: v1-completions-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/chat/completions [e2e-llm-inference-service] name: v1-chat-completions-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/responses [e2e-llm-inference-service] name: v1-responses-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/messages [e2e-llm-inference-service] name: v1-messages-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] name: v1-model-routing [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] name: v1-completions-publisher-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] name: v1-chat-completions-publisher-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] name: v1-responses-publisher-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] name: v1-messages-publisher-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] name: v1-publisher-path-catch-all [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] name: v1-catch-all-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] name: v1-catch-all-model-routing [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parentRefs: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66549' [e2e-llm-inference-service] uid: cec4ef32-b856-4a25-bb76-f64c765c15fb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/completions [e2e-llm-inference-service] name: v1-completions-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/chat/completions [e2e-llm-inference-service] name: v1-chat-completions-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/responses [e2e-llm-inference-service] name: v1-responses-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/messages [e2e-llm-inference-service] name: v1-messages-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] name: v1-model-routing [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] name: v1-completions-publisher-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] name: v1-chat-completions-publisher-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] name: v1-responses-publisher-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] name: v1-messages-publisher-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] name: v1-publisher-path-catch-all [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] name: v1-catch-all-path [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] name: v1-catch-all-model-routing [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66530' [e2e-llm-inference-service] uid: 2f241849-c10b-423c-9d5f-1eed1e42f42b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:50Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-10-30T23:28:53Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66483' [e2e-llm-inference-service] uid: 83225fe8-e789-466e-8bf7-8e5087ec3aa6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: tls-verification-test-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:52Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-10-30T23:28:53Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66299' [e2e-llm-inference-service] uid: b5f87d03-bf4d-4e43-810d-f82c98b6a50d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66535' [e2e-llm-inference-service] uid: c45dd1d6-3c19-4316-8fc7-10c7c4b70e1e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-inference-pool-ip-e386aca5.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66302' [e2e-llm-inference-service] uid: 29b2cbe2-b851-4663-9710-71260837a318 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66299' [e2e-llm-inference-service] uid: b5f87d03-bf4d-4e43-810d-f82c98b6a50d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66535' [e2e-llm-inference-service] uid: c45dd1d6-3c19-4316-8fc7-10c7c4b70e1e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-inference-pool-ip-e386aca5.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66302' [e2e-llm-inference-service] uid: 29b2cbe2-b851-4663-9710-71260837a318 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66299' [e2e-llm-inference-service] uid: b5f87d03-bf4d-4e43-810d-f82c98b6a50d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:58Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66535' [e2e-llm-inference-service] uid: c45dd1d6-3c19-4316-8fc7-10c7c4b70e1e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-inference-pool-ip-e386aca5.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66302' [e2e-llm-inference-service] uid: 29b2cbe2-b851-4663-9710-71260837a318 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"88b28efc-3090-4830-ac61-db8b42971220"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-10-30T23:28:49Z' [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 88b28efc-3090-4830-ac61-db8b42971220 [e2e-llm-inference-service] resourceVersion: '66292' [e2e-llm-inference-service] uid: c6e5747c-e89d-41c4-b04d-6f4003cf2d46 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-656f77bc69-6b9c8 [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:33:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 656f77bc69 [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] topology.kubernetes.io/region: us-east-2 [e2e-llm-inference-service] topology.kubernetes.io/zone: us-east-2a [e2e-llm-inference-service] timestamp: '2026-10-30T23:33:29Z' [e2e-llm-inference-service] window: 14.603s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 17822502n [e2e-llm-inference-service] memory: 21472Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 98910514n [e2e-llm-inference-service] memory: 2399628Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-577b5bd485-cjmrq [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:33:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 577b5bd485 [e2e-llm-inference-service] serving.kserve.io/llmisvc-revision: c7289d3e [e2e-llm-inference-service] topology.kubernetes.io/region: us-east-2 [e2e-llm-inference-service] topology.kubernetes.io/zone: us-east-2a [e2e-llm-inference-service] timestamp: '2026-10-30T23:33:39Z' [e2e-llm-inference-service] window: 31.564s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 101292390n [e2e-llm-inference-service] memory: 2405768Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5 [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] creationTimestamp: '2026-10-30T23:33:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69f7c679f8 [e2e-llm-inference-service] topology.kubernetes.io/region: us-east-2 [e2e-llm-inference-service] topology.kubernetes.io/zone: us-east-2a [e2e-llm-inference-service] timestamp: '2026-10-30T23:33:27Z' [e2e-llm-inference-service] window: 10.078s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 23232288n [e2e-llm-inference-service] memory: 30628Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-10-30T23:33:50.839257] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1453 Waiting: 3 pod(s) for tls-verification-test still terminating: ['tls-verification-test-kserve-656f77bc69-6b9c8', 'tls-verification-test-kserve-prefill-577b5bd485-cjmrq', 'tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5'] [e2e-llm-inference-service] assert not ['tls-verification-test-kserve-656f77bc69-6b9c8', 'tls-verification-test-kserve-prefill-577b5bd485-cjmrq', 'tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1453 Waiting: 1 pod(s) for tls-verification-test still terminating: ['tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5'] [e2e-llm-inference-service] assert not ['tls-verification-test-kserve-router-scheduler-69f7c679f8-x5lp5'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-10-30T23:34:41.261316] end - ✅ in 50.422s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_tls_resources] [2026-10-30T23:34:41.261464] end - ❌ 367.807s: 'NoneType' object has no attribute 'decode' [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:195 Skipping deletion of namespace e2e-test-llm-tls-resources-e4cad0c3 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:977: 250 warnings [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:977: PytestDeprecationWarning: The "scope" keyword argument to the asyncio marker has been deprecated. Please use the "loop_scope" argument instead. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn(PytestDeprecationWarning(_MARKER_SCOPE_KWARG_DEPRECATION_WARNING)) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-template-args-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-oci] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:236: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:270: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llmisvc_core_and_cluster_cpu_and_not_pvc_storage.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llmisvc_core_and_cluster_cpu_and_not_pvc_storage.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] - AttributeError: 'NoneType' object has no attribute 'decode' [e2e-llm-inference-service] ====== 1 failed, 44 passed, 8 skipped, 279 warnings in 4175.98s (1:09:35) ====== [e2e-llm-inference-service] make[1]: *** [Makefile.ocp.mk:85: run-e2e-ocp] Error 1 [e2e-llm-inference-service] make[1]: Leaving directory '/workspace/source' [e2e-llm-inference-service] make: *** [Makefile.ocp.mk:106: e2e-llmisvc-ocp] Error 2 [must-gather] [must-gather ] OUT 2026-10-30T23:40:54.808682035Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 14dd1418-000a-44c2-b6a2-41b3111578dc [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.22.20" [must-gather] ClusterOperators: [must-gather] clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-10-30T23:40:54.942468932Z namespace/openshift-must-gather-lzrz9 created [must-gather] [must-gather ] OUT 2026-10-30T23:40:54.961974846Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-brg42 created [must-gather] [must-gather ] OUT 2026-10-30T23:40:55.004880036Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:02.235947217Z [disk usage checker] Started [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:02.239281410Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:02.526403618Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:07.243724124Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.248151100Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.612860765Z E1030 23:41:12.612822 39 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.698243258Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.698792426Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.698792426Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.702401436Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.874212960Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.874292796Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.874292796Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:12.877540734Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.075741917Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.075784250Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.075784250Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.078913899Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.251011517Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.251055486Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.251055486Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.254366398Z Error getting logs from rhods-notebooks [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.435631073Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.435669692Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.435669692Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.438819581Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.607751527Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.607880842Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.607880842Z namespaces "istio-system" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.611043533Z Error getting logs from istio-system [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.778312166Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.778356439Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.778356439Z namespaces "knative-serving" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.781396445Z Error getting logs from knative-serving [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.951631288Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.951675004Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.951675004Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:13.954906089Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:14.251365028Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:14.552919849Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:14.697193630Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:14.782283578Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:14.785240497Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:14.926724823Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.011054887Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.013814138Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.156225963Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.241307674Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.244100318Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.387227866Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.472578730Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.475386882Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.616562053Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.701049370Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.703825762Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.845919380Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.928747969Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:15.931315156Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.071079553Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.154762320Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.157522787Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.302648326Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.388590902Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.391664772Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.535995374Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.625426799Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.628133940Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.772406635Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.856369696Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:16.859229905Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:17.001904706Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:17.085004821Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:17.087694419Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:17.252662035Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:22.257304088Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:27.170415433Z E1030 23:41:27.170368 491 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:27.261772609Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:32.266236132Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.175208671Z E1030 23:41:37.175168 491 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.231321362Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.271167387Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.319024966Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.322002012Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.467266086Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.552999494Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.556073048Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.699205802Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.784324444Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:37.787173228Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:42.275775890Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:47.280162290Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:47.873322324Z E1030 23:41:47.873278 594 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:47.931387886Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.015061461Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.017990587Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.161004385Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.248484906Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.251533345Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.396631375Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.483787867Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.486932797Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.628700756Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.715692987Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:48.718708709Z Error collecting info from [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:49.496889637Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:49.638308765Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:52.284558443Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:57.289274405Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:59.728203269Z E1030 23:41:59.728163 796 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:59.782611998Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-x675w] POD 2026-10-30T23:41:59.923113265Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:02.293772926Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:07.298257303Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:10.011935598Z E1030 23:42:10.011889 830 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:10.068490487Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:12.302831090Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:17.307266033Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:20.658983930Z E1030 23:42:20.658940 901 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:20.714716753Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:20.857541561Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:21.003814967Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:21.157020416Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:21.304983003Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:22.311696558Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:27.316202189Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:31.397465045Z E1030 23:42:31.397423 973 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:31.878432615Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.044553104Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.145751208Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.227766177Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.320699281Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.405003598Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.555729652Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.649121424Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.727075236Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:32.904953862Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:33.056837469Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:33.148544724Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:33.228846108Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:33.405243904Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:33.554929844Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:33.644439801Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:33.726678694Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:37.325208783Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:42.329820091Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:43.817744965Z E1030 23:42:43.817704 1076 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:43.901929887Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:44.053466447Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:44.160892734Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:44.240249116Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:44.415115905Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:44.566189458Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:44.655919279Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:44.735832466Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:44.905156831Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:45.057001909Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:45.145015964Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:45.226749660Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:45.401665338Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:45.558467828Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:45.652436072Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:45.737971268Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:47.334308475Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:52.338757235Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:55.826319894Z E1030 23:42:55.826275 1135 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:55.920655647Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:56.081732181Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:56.182735920Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:56.266119758Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:42:57.343345489Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:02.347934797Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:06.355959534Z E1030 23:43:06.355916 1157 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:06.440322239Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:06.599306575Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:06.701702742Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:06.785584405Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:06.995790997Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:07.164364606Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:07.260606623Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:07.342913437Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:07.352502308Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:07.517094472Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:07.667362401Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:07.763323882Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:07.848929383Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:08.020763915Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:08.177824037Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:08.290836161Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:08.381720937Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:08.560844890Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:08.710621815Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:08.799694639Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:08.880160848Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:09.053205425Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:09.207840650Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:09.299413841Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:09.387738755Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:09.563091995Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:09.724540696Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:09.877716402Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:09.960552400Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:10.131348721Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:10.977982187Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:11.300981016Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:11.468606738Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:11.645023560Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:12.357024128Z [disk usage checker] Volume usage percentage: current = 23 ; allowed = 70 [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:12.370143586Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:12.673193359Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:12.841871438Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:13.013218266Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:13.489782088Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:13.685330141Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:13.862356766Z Wrote inspect data to must-gather. [must-gather] [must-gather-x675w] POD 2026-10-30T23:43:13.903982712Z Caches written to disk [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:14.934218071Z waiting for gather to complete [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:14.950522571Z downloading gather output [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.356275773Z receiving incremental file list [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.396241098Z ./ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.396377121Z aggregated-discovery-api.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.396721927Z aggregated-discovery-apis.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.397979941Z event-filter.html [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.398891857Z timestamp [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.39905043Z version [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.409592764Z cluster-scoped-resources/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.409627775Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.409639515Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.409774088Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410028152Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410063213Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410153105Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410342468Z namespaces/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410356518Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.41041963Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410490521Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410568022Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410623953Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410689175Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410791016Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.410912668Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.41101386Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411079182Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411141453Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411192584Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411256195Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411357027Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411409528Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411474769Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411575901Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411631532Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411695603Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411853696Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.411978008Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.41208152Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412183122Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412273244Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412417036Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.41260423Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412673121Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412739042Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412794053Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412851454Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412939046Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.412985027Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413092569Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.41313984Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413205691Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413256592Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413318233Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413390614Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413434925Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413503596Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413544867Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413611258Z namespaces/kuadrant-system/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413663929Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413737981Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413794782Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413854453Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.413920064Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.414028756Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.414400503Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.414966193Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415015674Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415074615Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415130776Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415192557Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415296879Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.41534795Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415410401Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415511463Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415563264Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415624155Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415778768Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.415948121Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.416474441Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.416569793Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.423849517Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.42404003Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424323115Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424468158Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424512629Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424653632Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424695772Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424738453Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424782834Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424888016Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424929697Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.424973367Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.425018848Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.425070099Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.42510603Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.42511291Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.425148281Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino-557c8f8584-6kpnv.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.425234912Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.425242572Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.425246603Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.425279813Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.440574435Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.440642166Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.440665187Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.440721697Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/authorino-operator-66498fdf6c-b8722.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.44084298Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.44085649Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.44088004Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.440910191Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441119745Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441205066Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441249457Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441316118Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/dns-operator-controller-manager-68df894cb4-nk85t.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.44142131Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441430561Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441435701Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441476121Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441604984Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441691615Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441736636Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441799997Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin-66c4fb575d-pk9xx.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.441905539Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.44191817Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.44192313Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.44195894Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.442060062Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.442148744Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.442183424Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.442248036Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/kuadrant-operator-controller-manager-658c459c99-2n266.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.442366128Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.442373618Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.442377718Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.442422789Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453098765Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453179297Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453232768Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453299829Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador-limitador-7c9b87d574-ml7lt.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453396861Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453405341Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453410351Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453451432Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453562644Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453651195Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453697196Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.453761788Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/limitador-operator-controller-manager-69b7bb8c5c-fsdcp.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.45388121Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.4538941Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.45390082Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.45392616Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454097204Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454183865Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454222916Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454286267Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454336768Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454396679Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.45446479Z namespaces/openshift-ingress/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454538862Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454618393Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454677694Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454738275Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454798516Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.454917769Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.455179414Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.455425548Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.455478839Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.45554321Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.455626932Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.455671103Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.455774205Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.455828796Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.455904837Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.456013169Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.45606524Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.456125671Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.456574039Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.456697291Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.457761431Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.457877393Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.458576046Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.458727119Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.458959373Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459061395Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459113686Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459213738Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459251339Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.45932079Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459362411Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459469443Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459520574Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459583435Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459651886Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459708427Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459782939Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459792299Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459842189Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/istiod-openshift-gateway-7ddd9cfdc8-wr97s.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459977262Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459990232Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.459995332Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.460038043Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.472482292Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.472563394Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.472618175Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.472688146Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/openshift-ai-inference-openshift-default-86b669b864-2c7vh.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.472818468Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.472827829Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.472833129Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.472856709Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473016722Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473102134Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473144614Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473213656Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/openshift-ai-inference-openshift-default-86b669b864-5cx6f.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473319288Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473328828Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473335578Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473369319Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473503161Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473587703Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473636803Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473701365Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/openshift-ai-inference-openshift-default-86b669b864-8g5tm.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473809527Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473817687Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473822777Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.473882268Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474020501Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474105842Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474157213Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474219014Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/openshift-ai-inference-openshift-default-86b669b864-bv98j.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474329146Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474336007Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474339827Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474387777Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.47453268Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474621522Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474665113Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474728234Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/openshift-ai-inference-openshift-default-86b669b864-njf8w.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474837536Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474847186Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474852836Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.474916977Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475152431Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475252583Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475304654Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/openshift-ai-inference-openshift-default-86b669b864-nn2ft.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475410426Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475417326Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475421266Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475469287Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475676751Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475776063Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475826664Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/openshift-ai-inference-openshift-default-86b669b864-rd99c.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475954676Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475966206Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.475972237Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476012137Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476223581Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476325883Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476377414Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/openshift-ai-inference-openshift-default-86b669b864-t2dz9.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476486556Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476496536Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476502186Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476540747Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476675939Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476762411Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476802142Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476885923Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/openshift-ai-inference-openshift-default-86b669b864-wvd5p.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.476995045Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477002296Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477006425Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477050876Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477190519Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477272741Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477306471Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477353332Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/openshift-ai-inference-openshift-default-86b669b864-zctx5.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477443224Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477451414Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477457524Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477489505Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477639267Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477727529Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.47777312Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477824051Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router-default-6668ff59b7-vglrn.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477946743Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477959733Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477965983Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.477990754Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/current.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.478105566Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/previous.insecure.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.478171527Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/previous.log [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.478211048Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.478255829Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.47830462Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.47834997Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.513671441Z [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.513712901Z sent 3,638 bytes received 1,017,890 bytes 681,018.67 bytes/sec [must-gather] [must-gather-x675w] OUT 2026-10-30T23:43:15.513725032Z total size is 14,693,305 speedup is 14.38 [must-gather] [must-gather ] OUT 2026-10-30T23:43:15.636239608Z namespace/openshift-must-gather-lzrz9 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 14dd1418-000a-44c2-b6a2-41b3111578dc [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.22.20" [must-gather] ClusterOperators: [must-gather] clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-10-30T23:43:15.85848757Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7a095d2a811d6a837d5e67c9263fdef2d3f8cc926f849c48493e50f6c862a3b7 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 14dd1418-000a-44c2-b6a2-41b3111578dc [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.22.20" [must-gather] ClusterOperators: [must-gather] clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-10-30T23:43:15.950424723Z namespace/openshift-must-gather-dn4ld created [must-gather] [must-gather ] OUT 2026-10-30T23:43:15.969743138Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-frfv8 created [must-gather] [must-gather ] OUT 2026-10-30T23:43:16.023632361Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7a095d2a811d6a837d5e67c9263fdef2d3f8cc926f849c48493e50f6c862a3b7 created [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:17.601881575Z [disk usage checker] Started [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:17.605540184Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:17.853239347Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:18.010367067Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:18.116820647Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:18.233067734Z Gathering data for ns/default... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:18.600545422Z Gathering data for ns/openshift... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:19.067450835Z Gathering data for ns/kube-system... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:20.039664298Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:20.473167864Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:20.473202441Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:20.473202441Z namespaces "assisted-installer" not found [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:22.610469557Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:27.615030851Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:32.619805349Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:37.624591815Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.461462780Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.467515601Z INFO: Gathering HAProxy config files [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.474994192Z INFO: Collecting host service logs for crio [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.475355467Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.475728891Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.476108087Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.476577892Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.476947748Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.477285974Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.477603561Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.477968096Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.478328118Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.478712260Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.479111521Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.485072874Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.490853581Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.491324353Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.504911421Z Executing Istio gather script [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:40.635931719Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.100284025Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-xkcch [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.100284025Z insights-runtime-extractor-5rttv [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.100284025Z insights-runtime-extractor-6mwl7 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.100284025Z insights-runtime-extractor-6w8dk [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.186135391Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.590993170Z No resources found [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.747678961Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.783757785Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.915309332Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.937713363Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.944935346Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.944935346Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:41.966315292Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.035609100Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.035609100Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.047852207Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.063767358Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.106475506Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.186314451Z INFO: "nfd" not detected. Skipping. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.334230148Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.681079258Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.687659485Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:42.918967984Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.051241827Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.054796807Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.170341933Z No resources found [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.296187237Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.339955267Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.351175744Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.396646769Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.456069406Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.459354031Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.689498002Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.736099612Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.740377461Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:43.930663120Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.037050042Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.040155703Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.081337416Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.332378261Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.335704535Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.423300699Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.479636248Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.638376119Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.641387684Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.697454955Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.759330989Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.781225928Z INFO: LEGACY MODE [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.781267282Z INFO: Gathering ovn-kubernetes master data [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:44.907512290Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:45.604176253Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:45.604217411Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:46.087770510Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:46.916673095Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:47.770523877Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:51.020164264Z E1030 23:43:51.020124 283 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:51.593159390Z Gathering data for ns/openshift-cloud-credential-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:51.598389980Z E1030 23:43:51.598354 462 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:51.773680184Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:51.895707614Z E1030 23:43:51.895652 663 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:51.914853837Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.240336373Z E1030 23:43:52.240297 793 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.279355012Z E1030 23:43:52.279309 754 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.316751168Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.332079101Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.428568878Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.432733462Z INFO: 'previous-logs' folder not found on ip-10-0-128-97.us-east-2.compute.internal, skipping... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.595281659Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.678939522Z INFO: 'previous-logs' folder not found on ip-10-0-132-192.us-east-2.compute.internal, skipping... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.780148794Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.786030472Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.959786615Z INFO: 'previous-logs' folder not found on ip-10-0-138-251.us-east-2.compute.internal, skipping... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.959835161Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:52.959846283Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.020745143Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.022641743Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023197234Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 810 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 812 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 813 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 814 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 816 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 817 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 818 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 819 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 828 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 832 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 833 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023279608Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 835 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.023304651Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 836 is not a child of this shell [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.026238355Z tar: ovnk_database_store: Cannot stat: No such file or directory [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.031294514Z tar: Exiting with failure status due to previous errors [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.031878015Z INFO: Network log collection complete. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.074086411Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.242956397Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.473957193Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.528131392Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:53.778236968Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:54.013613809Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:54.249258096Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:54.484777609Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:54.754506193Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:55.000979228Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:55.004357547Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:55.249365400Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:55.285397658Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:55.450982315Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:55.587760396Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:55.636621740Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:56.392235353Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:56.915452040Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:56.998234636Z E1030 23:43:56.998187 1313 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:57.385830747Z Gathering data for ns/openshift-console... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:57.786080385Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:58.007161940Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:58.633147256Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:59.087452093Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:43:59.919594654Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:00.556585864Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:01.026506711Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:01.899345597Z E1030 23:44:01.899308 663 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:01.939063637Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:02.105576662Z tar: Removing leading `/' from member names [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:02.111380912Z INFO: HAProxy config collection complete. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:02.528380966Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:02.791127468Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:03.100753998Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:05.703357422Z E1030 23:44:05.703313 1615 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:05.742412733Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:06.031157117Z Collecting performance related data for node ip-10-0-138-251.us-east-2.compute.internal [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:06.038286936Z Collecting performance related data for node ip-10-0-128-97.us-east-2.compute.internal [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:06.046037971Z Collecting performance related data for node ip-10-0-132-192.us-east-2.compute.internal [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:06.525612519Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.005477266Z E1030 23:44:07.005429 1313 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.118756432Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.340741924Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.343829112Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.364443761Z Gathering data for ns/kserve... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.543790616Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.751951664Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.754741283Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:07.796170503Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:09.915004327Z Gathering data for ns/openshift-config... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:10.386827770Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:10.922519859Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:11.287489420Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:11.724476891Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:12.195649492Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:12.622290915Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:12.800958656Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:13.025883323Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:13.558764272Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:14.074657045Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:14.539590727Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:15.181446774Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:16.702349303Z E1030 23:44:16.702309 1779 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:17.432731380Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:17.805634168Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:17.840190364Z E1030 23:44:17.840151 1869 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:17.984142541Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:18.188793244Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:18.191674309Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:18.372793256Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:18.632246028Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:18.636932373Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:18.836074804Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:19.110422839Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:19.119974179Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:19.376198279Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:19.610750672Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:19.613583473Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:19.798815719Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:19.821511905Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.032405977Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.035382655Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.286891512Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.326685109Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.497172744Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.500329459Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.750213233Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.989167340Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:20.992424887Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:21.251786051Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:21.296791521Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:21.513679111Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:21.517528914Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:21.724529168Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:21.960740750Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:21.964534783Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:21.978442144Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:22.152071429Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:22.387556943Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:22.390044252Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:22.596718060Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:22.658455300Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:22.810357156Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:22.833615195Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:22.836480834Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.042777718Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.137732855Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.274645093Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.282609843Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.501506521Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.732839261Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.736561707Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.865934304Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:23.962694291Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:24.172828075Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:24.177135694Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:24.336592626Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:24.875585891Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:25.233563758Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:25.598768153Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:26.409460416Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:26.808528556Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:27.135561060Z E1030 23:44:27.135519 1901 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:27.254270719Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:27.789611929Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:27.815063639Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:29.040938333Z Wrote inspect data to must-gather. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:29.040984185Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:29.040984185Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:32.819801997Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:34.279001974Z E1030 23:44:34.278961 2336 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:34.382337444Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:37.538182312Z E1030 23:44:37.538149 2358 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:37.824476140Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:42.829217446Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:44.463258962Z E1030 23:44:44.463218 2379 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:44.576389676Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:44.579493233Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:44.756120691Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:44.949870335Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:44.952652335Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:47.834218179Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:52.033580183Z E1030 23:44:52.033540 2474 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:52.708641525Z Collecting kubelet logs for node ip-10-0-138-251.us-east-2.compute.internal [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:52.710532512Z Collecting kubelet logs for node ip-10-0-128-97.us-east-2.compute.internal [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:52.712925608Z Collecting kubelet logs for node ip-10-0-132-192.us-east-2.compute.internal [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:52.839270397Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:55.030550279Z E1030 23:44:55.030494 2512 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:55.149127918Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:55.348532906Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:55.351382456Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:55.523431225Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:55.717807219Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:55.720561577Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:55.898564130Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:56.097696904Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:56.100761737Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:56.275056277Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:44:57.844173192Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:02.823460830Z E1030 23:45:02.823417 2545 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:02.849615082Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:03.176291898Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-dn4ld namespace [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:03.182226550Z INFO: Node performance data collection complete. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:06.357005380Z E1030 23:45:06.356951 2658 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:06.474847656Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:06.604721246Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:06.803552065Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:07.371129899Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:07.658531343Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:07.854694773Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:07.860446780Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:12.859497549Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:17.864342528Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:17.944980809Z E1030 23:45:17.944943 2719 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:22.869357866Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:27.874053429Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:27.948435209Z E1030 23:45:27.948385 2719 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:27.990026886Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:28.189888711Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:28.697083000Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:28.980024822Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:29.174466943Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:32.878725409Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:37.883408884Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:39.567164019Z E1030 23:45:39.567120 2792 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:39.771840246Z Done executing Istio gather script [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:39.909359482Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-6hmfk] POD 2026-10-30T23:45:39.969178302Z Caches written to disk [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.002651481Z waiting for gather to complete [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.020034224Z downloading gather output [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.392570686Z receiving incremental file list [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.432803103Z ./ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.432947227Z aggregated-discovery-api.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.433290705Z aggregated-discovery-apis.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.434060504Z event-filter.html [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.44500864Z timestamp [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445101532Z version [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445212694Z cluster-scoped-resources/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445226775Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445234065Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445270826Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445391859Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/egressip-mark-annotation-validation.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445513092Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445615954Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.44583455Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.445906671Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.446019314Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/egressip-mark-annotation-validation-binding.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.446119437Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.44624709Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.446332102Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.446397053Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.446537297Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.446630239Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.447108721Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.447168332Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.447374877Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.447646824Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.44790922Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.448106745Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.451728193Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.454851559Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.455023343Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.455165516Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.45530028Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.455451413Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.455972836Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.45613114Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.456310944Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.457108764Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.45736318Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.457502453Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.457711138Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.458917308Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/backendtlspolicies.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.459207705Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.45944364Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.459631385Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.459808429Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.460163298Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.460315121Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.460532987Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.460811384Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.461511811Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.461653624Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.461785977Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.461947421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.462077194Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.462280199Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.462493805Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.463291164Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.463441457Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.463587971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.46559264Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterservingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.466133113Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.466306707Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.466478441Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.466763758Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.467078086Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.467579878Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.467727632Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.467879955Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.468029389Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.468151472Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.468294486Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.468423218Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.468600413Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.468759817Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.468919541Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.469207668Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.469359981Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.469516525Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/containerruntimeconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.469697809Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.470276674Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.470447988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.470591881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datagathers.insights.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.470801266Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.471028502Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.47177793Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.471933294Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.472141369Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.472284022Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.472459597Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.47261039Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.472816015Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.472987449Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.473129023Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.473277227Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.47342123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.473563663Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.473697957Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.473877981Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.474033005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.474177498Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.474341452Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.474735582Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.474942887Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.475355617Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.47548565Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.476343991Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.476496555Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.476637938Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.476771381Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.476958276Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.477211102Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.477468688Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.477623022Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.477756285Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.47796314Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.478133834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.478315959Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.478439812Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.478588985Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.482701905Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.483178907Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.483751541Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.483969086Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsdatagathers.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.48410825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.484251513Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.484383636Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.48452156Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.484625372Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.484746785Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.485557535Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.485853172Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.488791293Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.488937637Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.489080851Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.489211784Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.489339587Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.48947311Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.489593573Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.492474383Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.492800761Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.498690994Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.504421493Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.504633989Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.505064359Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.505250613Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.505437018Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.505573891Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.505771166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.506110424Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.506253308Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.506404371Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.506646907Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.506811421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.506968345Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.507075788Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.50717786Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.507325824Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.507481648Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.50758519Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.507715383Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.507838006Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.508015731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.508116553Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.508251836Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.508374579Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.508502612Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.50879854Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.508953954Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.50922082Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.509336693Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.509459206Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.509570128Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.511840773Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.512030368Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.512183352Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.512326045Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.512462389Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.512661844Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.512818847Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.513050993Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.513211407Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.515156504Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.515351319Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.515508193Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.515696437Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.515852441Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.516107717Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.516407495Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.51705466Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.517367398Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.517523942Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.517666105Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.517811318Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.518674399Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.518913655Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.519074279Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.520917864Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.52114825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.521313484Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.521428066Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.521606581Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.521920858Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.522101883Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.522325308Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.52281372Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.522975834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.523287112Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.523556148Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.523745183Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.523941967Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.524125532Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.524317567Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.524727886Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.524799268Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.524956032Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.525065515Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.525184078Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.52528677Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.525492635Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.525629948Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.525746181Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.525883785Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.526018088Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.526138281Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.526254294Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.526373747Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.5264978Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.526629183Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.526748186Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.526886489Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.527012252Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.527137915Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.527263818Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.527378101Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.527493264Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.527611617Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.52774225Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.527848673Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.528072998Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.528189881Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.528310974Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.528430827Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.528548319Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.528666752Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.insights.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.528786085Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.528931229Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.529051172Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.529178215Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.529278367Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.llm-d.ai.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.52941113Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.529529953Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.529646826Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.529766329Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.529909203Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.530035146Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.530147988Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.530265101Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.530386834Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.530510277Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53064381Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.530799574Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.530960748Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.531086781Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.531216764Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.531341197Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.531471221Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.531596044Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.531728297Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53185217Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.532010544Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.532159227Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53227908Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.532390023Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.532537136Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53268463Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.532824003Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.532974597Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53308212Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.533224033Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.533381967Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53349872Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.533589922Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.533709165Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.533841078Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.533984311Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.534110145Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.534226817Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53435404Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.534484874Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.534623017Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53473922Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.534874663Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.535033577Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53515646Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.535280873Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.535394676Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.535521539Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.535661672Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.535787075Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.535934829Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.536060642Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.536182105Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.536305688Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.536426291Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.536571244Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.536693507Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53681139Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.536957374Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.537074707Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.53720316Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.537334323Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.537466176Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.537593009Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.537719642Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.537839295Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.537973739Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.538084761Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.538195014Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.538262075Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.538276896Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.538345668Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/csr-hz247.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.538574953Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.538636335Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.5388438Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.539033524Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.539145807Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.539244519Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.539659189Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.539793283Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.539924716Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.540041119Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.540179702Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.540280535Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.540387817Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54048734Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.540594002Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.540696655Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.540811908Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.540941851Z cluster-scoped-resources/config.openshift.io/insightsdatagathers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541056543Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541166956Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541271259Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541377371Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541479504Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541585736Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541684809Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541832962Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.541910194Z cluster-scoped-resources/config.openshift.io/clusteroperators/cloud-credential.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.542031767Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54214844Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.542270583Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.542377586Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.542483158Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.542606701Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.542798196Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.542934579Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.543044842Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.543162765Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.543277687Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54339156Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.543559214Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.543673017Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54378108Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.543911283Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544019525Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544131218Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54423614Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544344863Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544460596Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544534558Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544594819Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544673091Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544720132Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544782294Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544830085Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.544984319Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54503023Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545114232Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545164473Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545238795Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545285616Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545363548Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545375468Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54543255Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545546072Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545650435Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545717307Z cluster-scoped-resources/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545726687Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.545779508Z cluster-scoped-resources/core/nodes/ip-10-0-128-97.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.546062575Z cluster-scoped-resources/core/nodes/ip-10-0-132-192.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.546210899Z cluster-scoped-resources/core/nodes/ip-10-0-138-251.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.546385263Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.546481565Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.546553557Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54667259Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.546776293Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.546881295Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.547001138Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54709941Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.547285075Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.547410858Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.5475198Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.547629193Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.547737816Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54790562Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.548031383Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.548144135Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.548258288Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.548371101Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.548527215Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.548645778Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.548760221Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.548879364Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.549000766Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.549061598Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.549117089Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.549224152Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.549361225Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.549474398Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.54957699Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.549688533Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.549880508Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550063042Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550170525Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550230306Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550241977Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550287038Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550353119Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550365Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550421111Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550551934Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550605266Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550668037Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550678017Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550748319Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.550858022Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551060646Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551142348Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551150059Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55119971Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551303482Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551310372Z cluster-scoped-resources/operator.openshift.io/cloudcredentials/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551357194Z cluster-scoped-resources/operator.openshift.io/cloudcredentials/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551419945Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551467526Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551559498Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55161357Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551753523Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551807115Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551918647Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.551973909Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552062251Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552115072Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552249375Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552294476Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552355788Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552401969Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55246448Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552513382Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552567923Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552620744Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552689356Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552741587Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.552809769Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.5528561Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553008624Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553059645Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553133227Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553185868Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55326624Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553315231Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553425254Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553432534Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553479225Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553615028Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55367044Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553796463Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.553936286Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554125261Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554255534Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554390638Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554524581Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554624213Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554675044Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554734856Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554839918Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.554967321Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.555142746Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55530062Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.555407592Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.555517015Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.555622087Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55572705Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.555830492Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.556003926Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55612796Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.556240282Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.556351335Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.556462788Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55657166Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-dns-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.556684463Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.556801096Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.556932969Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.557035182Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.557145204Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.557254767Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.557417131Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.557474142Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.557597235Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.557702118Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55779796Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.557997065Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.558114518Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.558232591Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.558349824Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.558458546Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/insights-operator-gather.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.55859298Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/insights-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.558703782Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/insights-runtime-extractor-role.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.558813125Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.558942708Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559054221Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559159163Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559264496Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559371579Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559474981Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559589564Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559699386Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-dns-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559812289Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.559940052Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560052475Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560173298Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560282081Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560393613Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560503816Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560614138Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560719451Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560830794Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.560961427Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561066109Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561172402Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561281305Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561344406Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561356677Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561422778Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.56151102Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561520261Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561564282Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561660644Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561670034Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561731496Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561810818Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.561872469Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-128-97.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562003912Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-132-192.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562108165Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-138-251.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562212988Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562263009Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562374401Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562425403Z host_service_logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562432063Z host_service_logs/masters/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562494094Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562594617Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562672459Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562757841Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.562937645Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563064928Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.56316124Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563249863Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563339975Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563428627Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563519499Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.56357667Z ingress_controllers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563589541Z ingress_controllers/default/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563593501Z ingress_controllers/default/router-default-6668ff59b7-vglrn/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563657143Z ingress_controllers/default/router-default-6668ff59b7-vglrn/haproxy.config [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563780906Z insights-data/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563814556Z istio/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.563880118Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.564022701Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.564532874Z istio/event-filter.html [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.565040186Z istio/timestamp [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.56518684Z istio/cluster-scoped-resources/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.56519654Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.56520151Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.565250521Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.565344964Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.565398775Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.565476087Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.565552849Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.565608Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.565798105Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/backendtlspolicies.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.566147103Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.567033405Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.567203409Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.567373933Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.567764172Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.567970167Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.568398408Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.569214658Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.569377282Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.569524115Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.569671049Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.569795392Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.569947885Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.570113049Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.570302054Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.570600611Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.570785946Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.571222216Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.57137561Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.571533854Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.571699718Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.571713568Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.571726019Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.571816631Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.571838011Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.571919903Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572032216Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.57219236Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572283312Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572341343Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572460966Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572640601Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572706283Z istio/namespaces/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572801345Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572879097Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.572967689Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.57301786Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.573110222Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.573161843Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.573268216Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.573518332Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.573766068Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.57381993Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.573901012Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.573999694Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.574050375Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.574159418Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.57423728Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.574290071Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.574403554Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.574490126Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.574551777Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.575026119Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.575146912Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.576283089Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.576380962Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.576665689Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.576807352Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577045408Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577160091Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577209652Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577307034Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577316944Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577359985Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577466598Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577517449Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577607682Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577659453Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577765865Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.577875948Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.57795318Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578010191Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578098254Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578105444Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578162985Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578256037Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578309128Z istio/namespaces/openshift-ingress/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578416831Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578511484Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578527034Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578586815Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/istiod-openshift-gateway-7ddd9cfdc8-wr97s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578714718Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578721458Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578731349Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.578802311Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591152941Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591234373Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591305864Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591380986Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/openshift-ai-inference-openshift-default-86b669b864-njf8w.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591512229Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591521929Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59153083Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591582271Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591791926Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59194063Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.591985091Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/openshift-ai-inference-openshift-default-86b669b864-nn2ft.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592095623Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592117374Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592125594Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592174016Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59238161Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592484623Z istio/namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592533994Z istio/namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router-default-6668ff59b7-vglrn.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592643377Z istio/namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592652037Z istio/namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592658547Z istio/namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592703558Z istio/namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592833301Z istio/namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592942894Z istio/namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.592984645Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593044337Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593103328Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593163669Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593222351Z machine_config_ondisk/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593230871Z machine_config_termination_logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593235871Z monitoring/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593240711Z monitoring/alertmanager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593285072Z monitoring/alertmanager/status.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593396075Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593477417Z monitoring/prometheus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593532038Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593644481Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.593725433Z monitoring/prometheus/rules.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59444406Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.594477961Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.594557153Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.595540287Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.595601579Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59565119Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.595771113Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.595883326Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.595994788Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59607748Z monitoring/prometheus/status/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.596132842Z monitoring/prometheus/status/config.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.596611733Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.596692575Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.596803378Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597039494Z namespaces/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597051884Z namespaces/cert-manager-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597056874Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597061644Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597093555Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597181807Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597195338Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597256519Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597462184Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59770765Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.597879974Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59812712Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598342625Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598395016Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-pjwqr.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598557451Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598609882Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598684023Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598739625Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598815417Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598878178Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59896417Z namespaces/cert-manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59897131Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.598977631Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.599041032Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.599201566Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.599385891Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.599535254Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.59975653Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600014066Z namespaces/default/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600061427Z namespaces/default/default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600146219Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.60019072Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600280582Z namespaces/default/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600325543Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600431096Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600532579Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600624281Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600694532Z namespaces/default/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600752544Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600841156Z namespaces/default/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.600908768Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.60100895Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601072832Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601132013Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601230356Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601340508Z namespaces/default/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601389869Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601515562Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601621845Z namespaces/default/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601878261Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.601994184Z namespaces/default/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.602092346Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.6022371Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.602422745Z namespaces/default/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.602504676Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.602564438Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.60265508Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.602700871Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.602767073Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.602817724Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.602946797Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603042529Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603098131Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603158022Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603216704Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603273075Z namespaces/default/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603376358Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603431459Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603441769Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603498511Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603654855Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.603839509Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604008703Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604227368Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604418953Z namespaces/default/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604461524Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604520725Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604582117Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604647579Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604663499Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604673619Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604735431Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.604913725Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.60510271Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.605249483Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.605463408Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.605671043Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.605679204Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.605685414Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.605756046Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.60593407Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.606116554Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.606271608Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.606488443Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.606736189Z namespaces/kserve/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.60678512Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.606878503Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.606924034Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.607013406Z namespaces/kserve/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.607209431Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.607317463Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.607555069Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.607738264Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.607809385Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.607871447Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.607978789Z namespaces/kserve/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608035571Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608152114Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608233976Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608290497Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.60839764Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608460491Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608476012Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608545713Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608655316Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608761219Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608930493Z namespaces/kserve/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.608987294Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.609284401Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.609412434Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.609676271Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.609771663Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.610096211Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.610247675Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.610504791Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.610621084Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.610679815Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.610804698Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.610851219Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.610962602Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611016823Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611141716Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611245089Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.61129796Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611364722Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611444824Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611491835Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611621078Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611667839Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611749951Z namespaces/kserve/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611856944Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611957656Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.611971746Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.612014627Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.612182492Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.612367966Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.61251877Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.612728765Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.6129465Z namespaces/kserve/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.61295807Z namespaces/kserve/pods/kserve-controller-manager-76d8b77fbb-9nwtn/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613010792Z namespaces/kserve/pods/kserve-controller-manager-76d8b77fbb-9nwtn/kserve-controller-manager-76d8b77fbb-9nwtn.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613131815Z namespaces/kserve/pods/kserve-controller-manager-76d8b77fbb-9nwtn/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613141945Z namespaces/kserve/pods/kserve-controller-manager-76d8b77fbb-9nwtn/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613146615Z namespaces/kserve/pods/kserve-controller-manager-76d8b77fbb-9nwtn/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613208967Z namespaces/kserve/pods/kserve-controller-manager-76d8b77fbb-9nwtn/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.61334974Z namespaces/kserve/pods/kserve-controller-manager-76d8b77fbb-9nwtn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613438732Z namespaces/kserve/pods/kserve-controller-manager-76d8b77fbb-9nwtn/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613506544Z namespaces/kserve/pods/llmisvc-controller-manager-86c86ffb6b-w7p6x/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613570715Z namespaces/kserve/pods/llmisvc-controller-manager-86c86ffb6b-w7p6x/llmisvc-controller-manager-86c86ffb6b-w7p6x.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613660638Z namespaces/kserve/pods/llmisvc-controller-manager-86c86ffb6b-w7p6x/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613668488Z namespaces/kserve/pods/llmisvc-controller-manager-86c86ffb6b-w7p6x/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613673128Z namespaces/kserve/pods/llmisvc-controller-manager-86c86ffb6b-w7p6x/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.613722019Z namespaces/kserve/pods/llmisvc-controller-manager-86c86ffb6b-w7p6x/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.654569581Z namespaces/kserve/pods/llmisvc-controller-manager-86c86ffb6b-w7p6x/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.654643763Z namespaces/kserve/pods/llmisvc-controller-manager-86c86ffb6b-w7p6x/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.654696665Z namespaces/kserve/pods/model-serving-api-56ddc4c448-mzrn8/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.654772006Z namespaces/kserve/pods/model-serving-api-56ddc4c448-mzrn8/model-serving-api-56ddc4c448-mzrn8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.65490392Z namespaces/kserve/pods/model-serving-api-56ddc4c448-mzrn8/server/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.65492021Z namespaces/kserve/pods/model-serving-api-56ddc4c448-mzrn8/server/server/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.65492716Z namespaces/kserve/pods/model-serving-api-56ddc4c448-mzrn8/server/server/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.654987942Z namespaces/kserve/pods/model-serving-api-56ddc4c448-mzrn8/server/server/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.655101654Z namespaces/kserve/pods/model-serving-api-56ddc4c448-mzrn8/server/server/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.655191207Z namespaces/kserve/pods/model-serving-api-56ddc4c448-mzrn8/server/server/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.655258928Z namespaces/kserve/pods/odh-model-controller-7f9f8cdc65-nq8qs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.65532779Z namespaces/kserve/pods/odh-model-controller-7f9f8cdc65-nq8qs/odh-model-controller-7f9f8cdc65-nq8qs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.655422982Z namespaces/kserve/pods/odh-model-controller-7f9f8cdc65-nq8qs/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.655432593Z namespaces/kserve/pods/odh-model-controller-7f9f8cdc65-nq8qs/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.655438833Z namespaces/kserve/pods/odh-model-controller-7f9f8cdc65-nq8qs/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.655514095Z namespaces/kserve/pods/odh-model-controller-7f9f8cdc65-nq8qs/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.679742813Z namespaces/kserve/pods/odh-model-controller-7f9f8cdc65-nq8qs/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.679819085Z namespaces/kserve/pods/odh-model-controller-7f9f8cdc65-nq8qs/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.679904617Z namespaces/kserve/pods/s3-init-zx9kx/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.679956528Z namespaces/kserve/pods/s3-init-zx9kx/s3-init-zx9kx.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680125482Z namespaces/kserve/pods/s3-init-zx9kx/download-hf-model/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680143973Z namespaces/kserve/pods/s3-init-zx9kx/download-hf-model/download-hf-model/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680159423Z namespaces/kserve/pods/s3-init-zx9kx/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680225265Z namespaces/kserve/pods/s3-init-zx9kx/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680340388Z namespaces/kserve/pods/s3-init-zx9kx/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.6804286Z namespaces/kserve/pods/s3-init-zx9kx/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680476631Z namespaces/kserve/pods/s3-init-zx9kx/s3-init/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680487011Z namespaces/kserve/pods/s3-init-zx9kx/s3-init/s3-init/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680491441Z namespaces/kserve/pods/s3-init-zx9kx/s3-init/s3-init/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680545403Z namespaces/kserve/pods/s3-init-zx9kx/s3-init/s3-init/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.68082469Z namespaces/kserve/pods/s3-init-zx9kx/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680937362Z namespaces/kserve/pods/s3-init-zx9kx/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.680980663Z namespaces/kserve/pods/seaweedfs-78bb9c6ff8-75jl9/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681037295Z namespaces/kserve/pods/seaweedfs-78bb9c6ff8-75jl9/seaweedfs-78bb9c6ff8-75jl9.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681132207Z namespaces/kserve/pods/seaweedfs-78bb9c6ff8-75jl9/seaweedfs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681145737Z namespaces/kserve/pods/seaweedfs-78bb9c6ff8-75jl9/seaweedfs/seaweedfs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681149777Z namespaces/kserve/pods/seaweedfs-78bb9c6ff8-75jl9/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681195948Z namespaces/kserve/pods/seaweedfs-78bb9c6ff8-75jl9/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681399084Z namespaces/kserve/pods/seaweedfs-78bb9c6ff8-75jl9/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681488006Z namespaces/kserve/pods/seaweedfs-78bb9c6ff8-75jl9/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681549177Z namespaces/kserve/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681612109Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681687911Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681749162Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681844434Z namespaces/kuadrant-system/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.681924666Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.682059209Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.682110771Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.682206483Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.682255814Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.682362317Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.682676555Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.682973672Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683020663Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683075874Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683151926Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683212728Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.68331456Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683384162Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683446013Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683552526Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683618957Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683633698Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683700659Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683813082Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.683936725Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.684044988Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.68414067Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.684193621Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.684374256Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.684511759Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.68494955Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.685064082Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.685493643Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.685647436Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.685881702Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686018556Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686068617Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686242971Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686282462Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686380214Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686432826Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686540878Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686652411Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686705642Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686778254Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686824165Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.686904397Z namespaces/kuadrant-system/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.68700602Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.687084662Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.687093512Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.687147313Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.687347508Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.687572233Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.687732557Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.687978813Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.688217369Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.68826803Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-r2pmc.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.688571748Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.688619779Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.688728282Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.688837454Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.688959427Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689016329Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.6890776Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689166192Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689216713Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689343366Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689523411Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689640754Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689746426Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689756716Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689799137Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino-557c8f8584-6kpnv.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.68991635Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689930841Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689935951Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.689967732Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697124026Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697172087Z namespaces/kuadrant-system/pods/authorino-557c8f8584-6kpnv/authorino/authorino/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697255769Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697277889Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/authorino-operator-66498fdf6c-b8722.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697409213Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697424413Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697437523Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697489214Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697686809Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697776691Z namespaces/kuadrant-system/pods/authorino-operator-66498fdf6c-b8722/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697840253Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.697937455Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/dns-operator-controller-manager-68df894cb4-nk85t.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698046088Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698054768Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698062458Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.69812345Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698249733Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698341155Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-68df894cb4-nk85t/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698387276Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698450608Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin-66c4fb575d-pk9xx.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698566081Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698574581Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698582171Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698652303Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698745755Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698837077Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-66c4fb575d-pk9xx/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.698899619Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.69896162Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/kuadrant-operator-controller-manager-658c459c99-2n266.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.699083383Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.699090643Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.699094463Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.699138395Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.706550924Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.706636097Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-658c459c99-2n266/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.706683578Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.706745879Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador-limitador-7c9b87d574-ml7lt.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.706844992Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.706854512Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.706877563Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.706922764Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707035906Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707123178Z namespaces/kuadrant-system/pods/limitador-limitador-7c9b87d574-ml7lt/limitador/limitador/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707192Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707255752Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/limitador-operator-controller-manager-69b7bb8c5c-fsdcp.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707360774Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707369855Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707379065Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707437036Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.70760932Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707696242Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-69b7bb8c5c-fsdcp/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707740924Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707802565Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707875577Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.707949489Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.70802523Z namespaces/kube-node-lease/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.708032351Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.708045271Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.708103322Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-128-97.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.708218275Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-132-192.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.708322558Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-138-251.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.70842829Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.70843805Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.708503972Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.708725467Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.708956863Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.709120737Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.709338322Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.709564398Z namespaces/kube-public/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.709574238Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.709584718Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.70964266Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.709804334Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.710009668Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.710167632Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.710397728Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.710658354Z namespaces/kube-system/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.710712156Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.710800268Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71087308Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.710966262Z namespaces/kube-system/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711018773Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711163077Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71128389Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711419643Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711467104Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711526816Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711583327Z namespaces/kube-system/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711646848Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711747831Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711827293Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.711901334Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712006727Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712068089Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712077889Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7121316Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-dupkg3rodwibbigr5fytpqctri.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712242213Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712353625Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712459388Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712625252Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712690634Z namespaces/kube-system/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.712755585Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71293936Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.713045202Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.713402681Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.713493213Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.713850292Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714015716Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714342984Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714424656Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714486547Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71458264Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714637451Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714708403Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714761964Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714883047Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.714988129Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.715057771Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.715119383Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.715195015Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.715262346Z namespaces/kube-system/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.715361099Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71542524Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.715434311Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.715503732Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.715661086Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71585035Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.716023375Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71624107Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.716465925Z namespaces/kube-system/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.716475466Z namespaces/kube-system/pods/global-pull-secret-syncer-bzxzm/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.716529407Z namespaces/kube-system/pods/global-pull-secret-syncer-bzxzm/global-pull-secret-syncer-bzxzm.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71663302Z namespaces/kube-system/pods/global-pull-secret-syncer-bzxzm/global-pull-secret-syncer/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71664012Z namespaces/kube-system/pods/global-pull-secret-syncer-bzxzm/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71665165Z namespaces/kube-system/pods/global-pull-secret-syncer-bzxzm/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.716708581Z namespaces/kube-system/pods/global-pull-secret-syncer-bzxzm/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71704835Z namespaces/kube-system/pods/global-pull-secret-syncer-bzxzm/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717137692Z namespaces/kube-system/pods/global-pull-secret-syncer-bzxzm/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717197483Z namespaces/kube-system/pods/global-pull-secret-syncer-hgsqv/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717258425Z namespaces/kube-system/pods/global-pull-secret-syncer-hgsqv/global-pull-secret-syncer-hgsqv.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717347507Z namespaces/kube-system/pods/global-pull-secret-syncer-hgsqv/global-pull-secret-syncer/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717356897Z namespaces/kube-system/pods/global-pull-secret-syncer-hgsqv/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717364697Z namespaces/kube-system/pods/global-pull-secret-syncer-hgsqv/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717426909Z namespaces/kube-system/pods/global-pull-secret-syncer-hgsqv/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717752347Z namespaces/kube-system/pods/global-pull-secret-syncer-hgsqv/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717844149Z namespaces/kube-system/pods/global-pull-secret-syncer-hgsqv/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717918281Z namespaces/kube-system/pods/global-pull-secret-syncer-xkt2l/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.717960342Z namespaces/kube-system/pods/global-pull-secret-syncer-xkt2l/global-pull-secret-syncer-xkt2l.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718050844Z namespaces/kube-system/pods/global-pull-secret-syncer-xkt2l/global-pull-secret-syncer/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718062204Z namespaces/kube-system/pods/global-pull-secret-syncer-xkt2l/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718067174Z namespaces/kube-system/pods/global-pull-secret-syncer-xkt2l/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718139886Z namespaces/kube-system/pods/global-pull-secret-syncer-xkt2l/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718449764Z namespaces/kube-system/pods/global-pull-secret-syncer-xkt2l/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718553976Z namespaces/kube-system/pods/global-pull-secret-syncer-xkt2l/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718611718Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-8cbpq/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718669789Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-8cbpq/kas-connection-checker-685895c7c9-8cbpq.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718767311Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-8cbpq/connection-checker/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718899615Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-8cbpq/connection-checker/connection-checker/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718915305Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-8cbpq/connection-checker/connection-checker/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.718985147Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-8cbpq/connection-checker/connection-checker/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719085819Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-8cbpq/connection-checker/connection-checker/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719173611Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-8cbpq/connection-checker/connection-checker/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719219312Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-b4mzj/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719283554Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-b4mzj/kas-connection-checker-685895c7c9-b4mzj.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719370336Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-b4mzj/connection-checker/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719381866Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-b4mzj/connection-checker/connection-checker/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719385927Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-b4mzj/connection-checker/connection-checker/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719425717Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-b4mzj/connection-checker/connection-checker/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.71952711Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-b4mzj/connection-checker/connection-checker/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719619522Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-b4mzj/connection-checker/connection-checker/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719658303Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-tsv8h/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719716795Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-tsv8h/kas-connection-checker-685895c7c9-tsv8h.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719803027Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-tsv8h/connection-checker/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719810497Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-tsv8h/connection-checker/connection-checker/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719817427Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-tsv8h/connection-checker/connection-checker/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.719894859Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-tsv8h/connection-checker/connection-checker/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.720011332Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-tsv8h/connection-checker/connection-checker/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.720097684Z namespaces/kube-system/pods/kas-connection-checker-685895c7c9-tsv8h/connection-checker/connection-checker/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.720164795Z namespaces/kube-system/pods/konnectivity-agent-cmh6h/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.720224707Z namespaces/kube-system/pods/konnectivity-agent-cmh6h/konnectivity-agent-cmh6h.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.720329909Z namespaces/kube-system/pods/konnectivity-agent-cmh6h/konnectivity-agent/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.72034347Z namespaces/kube-system/pods/konnectivity-agent-cmh6h/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.72035346Z namespaces/kube-system/pods/konnectivity-agent-cmh6h/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.720404211Z namespaces/kube-system/pods/konnectivity-agent-cmh6h/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.721644241Z namespaces/kube-system/pods/konnectivity-agent-cmh6h/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.721730643Z namespaces/kube-system/pods/konnectivity-agent-cmh6h/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.721796235Z namespaces/kube-system/pods/konnectivity-agent-ptpfs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.721857096Z namespaces/kube-system/pods/konnectivity-agent-ptpfs/konnectivity-agent-ptpfs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.721981739Z namespaces/kube-system/pods/konnectivity-agent-ptpfs/konnectivity-agent/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.72198993Z namespaces/kube-system/pods/konnectivity-agent-ptpfs/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.72199586Z namespaces/kube-system/pods/konnectivity-agent-ptpfs/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.722043351Z namespaces/kube-system/pods/konnectivity-agent-ptpfs/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.72325691Z namespaces/kube-system/pods/konnectivity-agent-ptpfs/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.723338832Z namespaces/kube-system/pods/konnectivity-agent-ptpfs/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.723398334Z namespaces/kube-system/pods/konnectivity-agent-wb9bv/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.723447895Z namespaces/kube-system/pods/konnectivity-agent-wb9bv/konnectivity-agent-wb9bv.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.723544237Z namespaces/kube-system/pods/konnectivity-agent-wb9bv/konnectivity-agent/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.723558148Z namespaces/kube-system/pods/konnectivity-agent-wb9bv/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.723568698Z namespaces/kube-system/pods/konnectivity-agent-wb9bv/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.723621649Z namespaces/kube-system/pods/konnectivity-agent-wb9bv/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.724817708Z namespaces/kube-system/pods/konnectivity-agent-wb9bv/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.724928531Z namespaces/kube-system/pods/konnectivity-agent-wb9bv/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.724982952Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725049014Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725149266Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal/haproxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725156567Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal/haproxy/haproxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725160457Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725217198Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725482055Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725569987Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-97.us-east-2.compute.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725607698Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725669859Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725749281Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal/haproxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725763101Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal/haproxy/haproxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725784722Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.725848033Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.726153161Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.726238683Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-192.us-east-2.compute.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.726276914Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.726339585Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.726423578Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal/haproxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.726433458Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal/haproxy/haproxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.726437488Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.726480479Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727045433Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727129365Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727177896Z namespaces/kube-system/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727243177Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727300139Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727308829Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.72736611Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727490333Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727536194Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727597516Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727673308Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727720679Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727834962Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727903323Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.727989865Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728044677Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728119919Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.72817761Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728280432Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728382745Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728492568Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728545919Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728609891Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728692862Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728750964Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728857226Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.728937219Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.72898758Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.729101772Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.729189265Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.729236786Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.730197789Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.730292512Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.730394814Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.730493796Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.730606109Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.730746103Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.730967708Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73104195Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731103051Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731183873Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731241524Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731340277Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731383308Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731491891Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731590763Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731650765Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731711666Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731783078Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731836369Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731922911Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.731968482Z namespaces/openshift-apiserver-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.732069955Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.732122466Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.732130456Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.732186498Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.732380092Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.732776262Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.732961846Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733172861Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733400697Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733440848Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73353965Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733562451Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733659753Z namespaces/openshift-apiserver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733710055Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733789356Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733836638Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73393814Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.733998982Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734099854Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734204207Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734303279Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734376391Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734419842Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734493963Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734543905Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734653157Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734722119Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73478275Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734901213Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.734998936Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.735041457Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73516805Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.735276553Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.735380655Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.735480127Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73558005Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.735722313Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.735914098Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73600837Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736049571Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736133803Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736185444Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736266326Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736303807Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73641618Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736517823Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736556493Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736620535Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736694457Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736752968Z namespaces/openshift-apiserver/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736889282Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736954033Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.736962303Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.737018225Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.737180379Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.737369693Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.737523847Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.737739172Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.737997509Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.73803936Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738115111Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738153092Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738220594Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738227854Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738231974Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738270825Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738349067Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738361677Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738430169Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738602043Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738786708Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.738957562Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7392875Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.739728891Z namespaces/openshift-authentication/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.739737441Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.739747161Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.739805792Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.739989807Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740179152Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740328635Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740570221Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740779956Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740787786Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740791916Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740825967Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740907239Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74092596Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.740964721Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.741113034Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.741291089Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.741428712Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.741631417Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.741880843Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.741939655Z namespaces/openshift-cloud-credential-operator/openshift-cloud-credential-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742009686Z namespaces/openshift-cloud-credential-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742051057Z namespaces/openshift-cloud-credential-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742107739Z namespaces/openshift-cloud-credential-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742151189Z namespaces/openshift-cloud-credential-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742232911Z namespaces/openshift-cloud-credential-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742310493Z namespaces/openshift-cloud-credential-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742395406Z namespaces/openshift-cloud-credential-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742442027Z namespaces/openshift-cloud-credential-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742500108Z namespaces/openshift-cloud-credential-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74258888Z namespaces/openshift-cloud-credential-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742641501Z namespaces/openshift-cloud-credential-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742751884Z namespaces/openshift-cloud-credential-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742813516Z namespaces/openshift-cloud-credential-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.742875807Z namespaces/openshift-cloud-credential-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74300237Z namespaces/openshift-cloud-credential-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.743070162Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.743131993Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.743200655Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-cloud-network-config-controller-aws.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.743323398Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-cloud-network-config-controller-azure.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.743437771Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-cloud-network-config-controller-gcp.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.743641506Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-cloud-network-config-controller-openstack.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.743772009Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-image-registry-azure.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.743946983Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-image-registry-gcs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.744055386Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-image-registry-ibmcos-powervs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.744175909Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-image-registry-ibmcos.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.744290012Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-image-registry-openstack.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.744500947Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-image-registry.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.744620349Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-ingress-azure.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.744737172Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-ingress-gcp.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.744839565Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-ingress-ibmcloud.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.744975548Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-ingress-powervs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.745094961Z namespaces/openshift-cloud-credential-operator/cloudcredential.openshift.io/credentialsrequests/openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.745159433Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.745166793Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.745220994Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.745323507Z namespaces/openshift-cloud-credential-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.745378998Z namespaces/openshift-cloud-credential-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74588027Z namespaces/openshift-cloud-credential-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746017283Z namespaces/openshift-cloud-credential-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746142326Z namespaces/openshift-cloud-credential-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746246389Z namespaces/openshift-cloud-credential-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746348822Z namespaces/openshift-cloud-credential-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746485505Z namespaces/openshift-cloud-credential-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74669557Z namespaces/openshift-cloud-credential-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746769992Z namespaces/openshift-cloud-credential-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746830493Z namespaces/openshift-cloud-credential-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746927166Z namespaces/openshift-cloud-credential-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.746977237Z namespaces/openshift-cloud-credential-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747057499Z namespaces/openshift-cloud-credential-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74710709Z namespaces/openshift-cloud-credential-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747212112Z namespaces/openshift-cloud-credential-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747314265Z namespaces/openshift-cloud-credential-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747388397Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747439828Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74753305Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747589062Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747663094Z namespaces/openshift-cloud-credential-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747716975Z namespaces/openshift-cloud-credential-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747822147Z namespaces/openshift-cloud-credential-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74793776Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.747958341Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.748010742Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.748172256Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74836096Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.748513044Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.748732949Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.748946145Z namespaces/openshift-cloud-credential-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.748991826Z namespaces/openshift-cloud-credential-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749067808Z namespaces/openshift-cloud-credential-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749118829Z namespaces/openshift-cloud-credential-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749246262Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749304533Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749384855Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749422096Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749493188Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749534619Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749645782Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749744984Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749848397Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.749925858Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.74997654Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750039251Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750104863Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750210445Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750258406Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750320568Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750427711Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750469502Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750475902Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750547444Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750646556Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750699317Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75082053Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.750945923Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.751058346Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75123231Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.751332773Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.751495977Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75162269Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.751700901Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.751761363Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.751834655Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.751901886Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.751984348Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75203574Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752106542Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752157763Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752259235Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752368078Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752418409Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752482711Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752535042Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752599874Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752711626Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752777098Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.752791108Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75285701Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.753047044Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.753237409Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.753390503Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.753599508Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.753823493Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.753883235Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754044789Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754055209Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75410727Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754165672Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754204953Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754264244Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754311255Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754398637Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754439218Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75450091Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754548371Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754615983Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754657204Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754792147Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754876989Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.754996612Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755030893Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755083484Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755125395Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755177196Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755267388Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75532539Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755373171Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755445823Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755498384Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755507904Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755552395Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755643437Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755727079Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755841282Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755957975Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.755993466Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.756101468Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75617788Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.756516269Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.756593991Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.756804456Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75697948Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757097693Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757156954Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757209655Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757251627Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757306418Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757333238Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75738951Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757440561Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757490562Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757573214Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757644816Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757692037Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757736128Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75779482Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757853211Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.757956564Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.758033535Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.758041766Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.758082147Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75823536Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.758417825Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.758577529Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.758878106Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759101021Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759253275Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759297126Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/aws-ebs-csi-driver-node-5z2tl.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759401519Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-driver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759411539Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-driver/csi-driver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759417509Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-driver/csi-driver/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75944919Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759553112Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759621364Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759664825Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-liveness-probe/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759674915Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759681646Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759690706Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759791208Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.75985687Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759909111Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-node-driver-registrar/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759922341Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759930392Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.759985943Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760086295Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760149897Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-5z2tl/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760197498Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760242399Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/aws-ebs-csi-driver-node-849dk.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760342982Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-driver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760353502Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-driver/csi-driver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760364032Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-driver/csi-driver/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760414563Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760507815Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760573317Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760615838Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-liveness-probe/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760625589Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760631609Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760641209Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760742371Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760806543Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760836563Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-node-driver-registrar/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760843174Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760851244Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.760941096Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761029958Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76110192Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-849dk/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761144921Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761181632Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/aws-ebs-csi-driver-node-h8xmf.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761276034Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-driver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761285514Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-driver/csi-driver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761289545Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-driver/csi-driver/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761343966Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761431268Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76150077Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761543401Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-liveness-probe/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761550631Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761556671Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761596502Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761697684Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761760656Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761792867Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-node-driver-registrar/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761801827Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761810627Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761878169Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.761982882Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762043923Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-h8xmf/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762058843Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762110485Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762175076Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762185726Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762212847Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762283399Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762315639Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762392791Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762406802Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762476843Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762493954Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762594666Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762647347Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762712819Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762722709Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7627616Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762822172Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762831412Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.762910984Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.763073538Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.763249202Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.763383585Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76359391Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.763836696Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.763917308Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76398145Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764018481Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764086333Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764124554Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764297818Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76437789Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764459131Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764704457Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764744589Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764832521Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764873212Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764949974Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.764989735Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765080537Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765124788Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765162889Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765246201Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765291882Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765299882Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765336843Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765407165Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.765453476Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.766073211Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.766162233Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.766310967Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.766387049Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.766598643Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.766754507Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.766921421Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.766997993Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767053574Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767114146Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767193838Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767245049Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76729472Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767355942Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767436054Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767525376Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767611338Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767640729Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76769512Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767756642Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767774352Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767856814Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767921476Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.767989287Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768006738Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768140851Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768210203Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768225693Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768264214Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768423728Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768605622Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768740276Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.768984541Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769200377Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769213517Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7wqr7/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769244288Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7wqr7/tuned-7wqr7.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76934186Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7wqr7/tuned/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769352101Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7wqr7/tuned/tuned/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769370131Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7wqr7/tuned/tuned/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769431292Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7wqr7/tuned/tuned/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769546335Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7wqr7/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769620367Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7wqr7/tuned/tuned/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769670568Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ggnd6/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.76972643Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ggnd6/tuned-ggnd6.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769796771Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ggnd6/tuned/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769804352Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ggnd6/tuned/tuned/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769810282Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ggnd6/tuned/tuned/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.769878893Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ggnd6/tuned/tuned/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770039857Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ggnd6/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770103459Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ggnd6/tuned/tuned/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.77016209Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ht6mr/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770197871Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ht6mr/tuned-ht6mr.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770285783Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ht6mr/tuned/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770292783Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ht6mr/tuned/tuned/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770306994Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ht6mr/tuned/tuned/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770348785Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ht6mr/tuned/tuned/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770466317Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ht6mr/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7705607Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-ht6mr/tuned/tuned/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770600371Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770664422Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770722564Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770767085Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770822106Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770838386Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.770904488Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-128-97.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771022171Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-132-192.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771106143Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-138-251.us-east-2.compute.internal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771202355Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771250407Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771348809Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7714015Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771455121Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771493093Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771564094Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771617185Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771714288Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.771936343Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772056036Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772088827Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772142438Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772182999Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772246641Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772329573Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772391534Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772442805Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772520178Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772598789Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772668881Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.772935748Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.77301977Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773185024Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773265136Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773393799Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773522952Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773713807Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773765808Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773802819Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773875811Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.773929462Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774001463Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774035844Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774124806Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774202088Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7742676Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774308431Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774362832Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774397103Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774489725Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774506036Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774605818Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774621118Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774722011Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774761682Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774792513Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.774846974Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775017608Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775199292Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775337906Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775541481Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775733005Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775749226Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775788677Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator-6ddb6c8b94-r6fws.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775880219Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator-watch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775898249Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.77590437Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.775956091Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.776051523Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.776126585Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.776184167Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.776197717Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.776205217Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.776225528Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.776956115Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777031757Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6ddb6c8b94-r6fws/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777065468Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777105729Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7771678Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777193441Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777265963Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777309004Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777385246Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777414637Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777490998Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777520819Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777611631Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777711854Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777804656Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777845017Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.777914899Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.77797269Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778026531Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778100953Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778152364Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778194095Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778276297Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778330409Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778337209Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.77837977Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778464792Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778549284Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778583255Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.778706408Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.77878846Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.77919896Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.779275111Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.779408955Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.779568539Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.779707472Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.779781004Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.779830185Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.779906447Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.779941658Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780002629Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.78003916Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780104912Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780140733Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780227855Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780300536Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780374548Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780408729Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780467991Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780519352Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780628224Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780662455Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780727237Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780762958Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.78085372Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.780990203Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781010744Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781046645Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781197718Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781376973Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781519706Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781732601Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781939806Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781957307Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-58c9f5f46d-tw8c8/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.781977247Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-58c9f5f46d-tw8c8/volume-data-source-validator-58c9f5f46d-tw8c8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782056409Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-58c9f5f46d-tw8c8/volume-data-source-validator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782067009Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-58c9f5f46d-tw8c8/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782073819Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-58c9f5f46d-tw8c8/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.78210262Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-58c9f5f46d-tw8c8/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782199773Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-58c9f5f46d-tw8c8/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782278135Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-58c9f5f46d-tw8c8/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782295645Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782349736Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782427358Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782532981Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782584422Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782653374Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782701455Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.782839938Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.78291206Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783011652Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783056634Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783133285Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783175546Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783260258Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7833039Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783391952Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783464543Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783564906Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783611117Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783654238Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783710169Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783755831Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783857933Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783931595Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.783989236Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784062038Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784117599Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784128049Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784185411Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784308674Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784359265Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784460788Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.78456074Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784664662Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784743235Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.784838317Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.78499311Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785197115Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785317568Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785364379Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785427011Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785475002Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785547704Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785582865Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785670527Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785767129Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.78581233Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785872302Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785943733Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.785991465Z namespaces/openshift-cluster-version/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.786082247Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.786153659Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.786163249Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7862265Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.786382254Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.786579519Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.786735363Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787065731Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787355328Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787402449Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787470501Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787509612Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787612124Z namespaces/openshift-config-managed/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787659105Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787724437Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787746848Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787829639Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787869971Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.787974243Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788053695Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788132077Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788189088Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788234439Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788282441Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788322311Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788403233Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788461855Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788516946Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788592188Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788655999Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.788701331Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.791745435Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.791806586Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.791928959Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792000561Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792086763Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792250087Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.79236371Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.79239649Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792455942Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792553874Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792582335Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792634506Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792675577Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792718808Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792753409Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.79279257Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.792932234Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793011685Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793067797Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793148519Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.79321099Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793248971Z namespaces/openshift-config-managed/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793333563Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793377564Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793388594Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793485277Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793645231Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.793911157Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.794189944Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.79443098Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.794651005Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.794696216Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.794760928Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.794771998Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.794809859Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.794878241Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.794983153Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795035865Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795095126Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795129267Z namespaces/openshift-config-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795183358Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795197829Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795231719Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795294421Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795308471Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795354292Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795515856Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795721781Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.795882665Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7961003Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796332636Z namespaces/openshift-config/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796382597Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.7965057Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796544921Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796598792Z namespaces/openshift-config/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796635353Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796717066Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796795487Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796874739Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796951771Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.796990602Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797039023Z namespaces/openshift-config/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797069914Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797154616Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797200297Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797231158Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.79731605Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797399812Z namespaces/openshift-config/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797432403Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797554356Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797632308Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797711179Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797789451Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.797873214Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798022857Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798160941Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798295914Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798340985Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798409587Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798445957Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798504199Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.79854848Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798616561Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798651822Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798742855Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798818077Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.798875858Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.79894128Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799001711Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799049092Z namespaces/openshift-config/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799131734Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799183815Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799201926Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799271688Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799413711Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799603876Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.799742039Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800059147Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800236621Z namespaces/openshift-config/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800280992Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800331973Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800376164Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800475117Z namespaces/openshift-console-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800510788Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800721223Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800757394Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800844136Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800897307Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.800996879Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801142383Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801255826Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801311657Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801347408Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80141372Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801467051Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801562173Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801598144Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801641505Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801721967Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801789129Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801796299Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80184836Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801947302Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.801986093Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.802494436Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.802597688Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.802853675Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803028979Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803159172Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803293055Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80349308Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803559572Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803596233Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803655274Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803687875Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803757746Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803803138Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80390099Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.803987982Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804035853Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804089854Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804150116Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804199437Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804265729Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80430813Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804357811Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804401352Z namespaces/openshift-console-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804484804Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804537645Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804553556Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.804601887Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80474961Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.805015357Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80514253Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.805343285Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.805539Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80554702Z namespaces/openshift-console-operator/pods/console-operator-7cd56b8dbc-dd7n4/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.805586631Z namespaces/openshift-console-operator/pods/console-operator-7cd56b8dbc-dd7n4/console-operator-7cd56b8dbc-dd7n4.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.805694564Z namespaces/openshift-console-operator/pods/console-operator-7cd56b8dbc-dd7n4/console-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.805707024Z namespaces/openshift-console-operator/pods/console-operator-7cd56b8dbc-dd7n4/console-operator/console-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.805713734Z namespaces/openshift-console-operator/pods/console-operator-7cd56b8dbc-dd7n4/console-operator/console-operator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.805723904Z namespaces/openshift-console-operator/pods/console-operator-7cd56b8dbc-dd7n4/console-operator/console-operator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806263837Z namespaces/openshift-console-operator/pods/console-operator-7cd56b8dbc-dd7n4/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806313508Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806345249Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806412411Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806447962Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806503403Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806511873Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806541344Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806580295Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806734349Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.806936744Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.807076957Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.807299492Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.807513318Z namespaces/openshift-console/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.807555219Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.807791184Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.807833556Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.807919128Z namespaces/openshift-console/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.807963179Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808044861Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808209905Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808528932Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808568013Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808614715Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808669626Z namespaces/openshift-console/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808717807Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808794229Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80885127Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808898761Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.808996454Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.809047185Z namespaces/openshift-console/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.809099386Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.809580958Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.80966441Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.809929686Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810007848Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810196773Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810323676Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810536171Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810598093Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810631973Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810705455Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810742206Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810815618Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.810849309Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811014903Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811098405Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811155296Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811199987Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811263149Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81129761Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811361211Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811403672Z namespaces/openshift-console/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811491024Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811545716Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811554036Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811594797Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81174102Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.811940095Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812074848Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812276353Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812475598Z namespaces/openshift-console/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812484678Z namespaces/openshift-console/pods/console-6b88dcb6c8-v2bvm/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812519159Z namespaces/openshift-console/pods/console-6b88dcb6c8-v2bvm/console-6b88dcb6c8-v2bvm.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812618322Z namespaces/openshift-console/pods/console-6b88dcb6c8-v2bvm/console/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812629662Z namespaces/openshift-console/pods/console-6b88dcb6c8-v2bvm/console/console/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812636472Z namespaces/openshift-console/pods/console-6b88dcb6c8-v2bvm/console/console/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812682343Z namespaces/openshift-console/pods/console-6b88dcb6c8-v2bvm/console/console/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812766065Z namespaces/openshift-console/pods/console-6b88dcb6c8-v2bvm/console/console/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812838687Z namespaces/openshift-console/pods/console-6b88dcb6c8-v2bvm/console/console/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812856467Z namespaces/openshift-console/pods/downloads-6dd789c8b7-7r6gx/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.812927889Z namespaces/openshift-console/pods/downloads-6dd789c8b7-7r6gx/downloads-6dd789c8b7-7r6gx.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813066953Z namespaces/openshift-console/pods/downloads-6dd789c8b7-7r6gx/download-server/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813080363Z namespaces/openshift-console/pods/downloads-6dd789c8b7-7r6gx/download-server/download-server/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813087273Z namespaces/openshift-console/pods/downloads-6dd789c8b7-7r6gx/download-server/download-server/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813105533Z namespaces/openshift-console/pods/downloads-6dd789c8b7-7r6gx/download-server/download-server/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813457952Z namespaces/openshift-console/pods/downloads-6dd789c8b7-7r6gx/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813518534Z namespaces/openshift-console/pods/downloads-6dd789c8b7-7r6gx/download-server/download-server/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813573055Z namespaces/openshift-console/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813611256Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813696448Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813731939Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813826311Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813884172Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.813970545Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814010666Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814102568Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814132229Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814218611Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814300223Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814396985Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814434576Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814483227Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814555939Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81459067Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814691602Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814788034Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814835356Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.814947588Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81503306Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815079181Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815199164Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815282986Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815380789Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815463771Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815546613Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815697827Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815911632Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.815998604Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816046295Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816101266Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816132997Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816219639Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81625745Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816368153Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816448135Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816630259Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8166686Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816736422Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816755542Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816846084Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816984838Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.816999198Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.817041929Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.817196543Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.817383857Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.817518961Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.817747006Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.817948091Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.817989562Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818038793Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818081705Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818168667Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818216178Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818266519Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81831157Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818353511Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818389642Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818472884Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818581927Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818663239Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81870241Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818748381Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818812542Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818878234Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818977576Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.818992687Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819049068Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81912712Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819189872Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819201322Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819233182Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819317004Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819357345Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819483589Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.81955643Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819642472Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819718234Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.819800336Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8199561Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820127604Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820175145Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820224597Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820277868Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820314799Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820390331Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820424862Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820514944Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820601346Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820743869Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.82078363Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820831241Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820874452Z namespaces/openshift-controller-manager/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.820992475Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.821024506Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.821036426Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.821078797Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.821229601Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.821418646Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.821546869Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.821748134Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.822673316Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.822696287Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.822760838Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.822794569Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.822918402Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.822958663Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823031905Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823078996Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823122437Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823164288Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.82325003Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823323882Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823407384Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823453285Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823494956Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823541627Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823586288Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.82366139Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823721142Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823767573Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823844935Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.823973858Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824006309Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824122091Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824201933Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824280755Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824362227Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824439649Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824562512Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824739306Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824792577Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824844959Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824914411Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.824967372Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825031323Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825070234Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825159266Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825245079Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.82530196Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825355151Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825394822Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825430663Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825509085Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825525855Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825586897Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825625788Z namespaces/openshift-dns-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.82571311Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825765341Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825804482Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies/dns-operator-allow.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825922295Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies/dns-operator-deny-all.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825980706Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.825995097Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.826024407Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.826249113Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.826446308Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.826597372Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.826814567Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827035532Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827070233Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827153375Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827225737Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827302709Z namespaces/openshift-dns/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.82735183Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827417431Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827498643Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827576665Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827614186Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827833241Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.827967375Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828054887Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828121788Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.82816767Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828241181Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828275952Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828378245Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828448546Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828483877Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8285813Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828655941Z namespaces/openshift-dns/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828695492Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828813205Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.828916498Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.829165204Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.829246306Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.829519623Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.829641715Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.829876641Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.829980244Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830015384Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830077606Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830110327Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830158378Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830205859Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830295941Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830373343Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830418734Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830458225Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830522737Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830560428Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830617739Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83066487Z namespaces/openshift-dns/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830745462Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830801224Z namespaces/openshift-dns/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830839515Z namespaces/openshift-dns/networking.k8s.io/networkpolicies/dns-default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.830961458Z namespaces/openshift-dns/networking.k8s.io/networkpolicies/openshift-dns-deny-all.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.831018499Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.831028139Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8310806Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.831237144Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.831416398Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.831551782Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.831748926Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.831983522Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.831994643Z namespaces/openshift-dns/pods/dns-default-6wrrt/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832034153Z namespaces/openshift-dns/pods/dns-default-6wrrt/dns-default-6wrrt.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832123596Z namespaces/openshift-dns/pods/dns-default-6wrrt/dns/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832130386Z namespaces/openshift-dns/pods/dns-default-6wrrt/dns/dns/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832139076Z namespaces/openshift-dns/pods/dns-default-6wrrt/dns/dns/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832190657Z namespaces/openshift-dns/pods/dns-default-6wrrt/dns/dns/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83230205Z namespaces/openshift-dns/pods/dns-default-6wrrt/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832366781Z namespaces/openshift-dns/pods/dns-default-6wrrt/dns/dns/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832421533Z namespaces/openshift-dns/pods/dns-default-6wrrt/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832434203Z namespaces/openshift-dns/pods/dns-default-6wrrt/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832439383Z namespaces/openshift-dns/pods/dns-default-6wrrt/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832474704Z namespaces/openshift-dns/pods/dns-default-6wrrt/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832575027Z namespaces/openshift-dns/pods/dns-default-6wrrt/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832648358Z namespaces/openshift-dns/pods/dns-default-6wrrt/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83270138Z namespaces/openshift-dns/pods/dns-default-96mpc/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832737681Z namespaces/openshift-dns/pods/dns-default-96mpc/dns-default-96mpc.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832821763Z namespaces/openshift-dns/pods/dns-default-96mpc/dns/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832829133Z namespaces/openshift-dns/pods/dns-default-96mpc/dns/dns/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832840113Z namespaces/openshift-dns/pods/dns-default-96mpc/dns/dns/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.832930715Z namespaces/openshift-dns/pods/dns-default-96mpc/dns/dns/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833051278Z namespaces/openshift-dns/pods/dns-default-96mpc/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83311488Z namespaces/openshift-dns/pods/dns-default-96mpc/dns/dns/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833154761Z namespaces/openshift-dns/pods/dns-default-96mpc/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833164211Z namespaces/openshift-dns/pods/dns-default-96mpc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833170091Z namespaces/openshift-dns/pods/dns-default-96mpc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833223492Z namespaces/openshift-dns/pods/dns-default-96mpc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833316095Z namespaces/openshift-dns/pods/dns-default-96mpc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833391597Z namespaces/openshift-dns/pods/dns-default-96mpc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833424647Z namespaces/openshift-dns/pods/dns-default-wghlv/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833462038Z namespaces/openshift-dns/pods/dns-default-wghlv/dns-default-wghlv.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83354365Z namespaces/openshift-dns/pods/dns-default-wghlv/dns/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83355137Z namespaces/openshift-dns/pods/dns-default-wghlv/dns/dns/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833555461Z namespaces/openshift-dns/pods/dns-default-wghlv/dns/dns/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833591061Z namespaces/openshift-dns/pods/dns-default-wghlv/dns/dns/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833681604Z namespaces/openshift-dns/pods/dns-default-wghlv/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833754545Z namespaces/openshift-dns/pods/dns-default-wghlv/dns/dns/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833788666Z namespaces/openshift-dns/pods/dns-default-wghlv/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833798856Z namespaces/openshift-dns/pods/dns-default-wghlv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833803067Z namespaces/openshift-dns/pods/dns-default-wghlv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.833836047Z namespaces/openshift-dns/pods/dns-default-wghlv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83395779Z namespaces/openshift-dns/pods/dns-default-wghlv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834022052Z namespaces/openshift-dns/pods/dns-default-wghlv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834072133Z namespaces/openshift-dns/pods/node-resolver-cf4tj/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834113504Z namespaces/openshift-dns/pods/node-resolver-cf4tj/node-resolver-cf4tj.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834201026Z namespaces/openshift-dns/pods/node-resolver-cf4tj/dns-node-resolver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834211366Z namespaces/openshift-dns/pods/node-resolver-cf4tj/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834220967Z namespaces/openshift-dns/pods/node-resolver-cf4tj/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834262158Z namespaces/openshift-dns/pods/node-resolver-cf4tj/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83435251Z namespaces/openshift-dns/pods/node-resolver-cf4tj/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834421951Z namespaces/openshift-dns/pods/node-resolver-cf4tj/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834470813Z namespaces/openshift-dns/pods/node-resolver-qqwss/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834526024Z namespaces/openshift-dns/pods/node-resolver-qqwss/node-resolver-qqwss.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834706729Z namespaces/openshift-dns/pods/node-resolver-qqwss/dns-node-resolver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834714829Z namespaces/openshift-dns/pods/node-resolver-qqwss/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834721429Z namespaces/openshift-dns/pods/node-resolver-qqwss/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83477957Z namespaces/openshift-dns/pods/node-resolver-qqwss/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834883093Z namespaces/openshift-dns/pods/node-resolver-qqwss/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834955575Z namespaces/openshift-dns/pods/node-resolver-qqwss/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.834992485Z namespaces/openshift-dns/pods/node-resolver-wqsgd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835033966Z namespaces/openshift-dns/pods/node-resolver-wqsgd/node-resolver-wqsgd.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835102328Z namespaces/openshift-dns/pods/node-resolver-wqsgd/dns-node-resolver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835109178Z namespaces/openshift-dns/pods/node-resolver-wqsgd/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835115448Z namespaces/openshift-dns/pods/node-resolver-wqsgd/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8351612Z namespaces/openshift-dns/pods/node-resolver-wqsgd/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835251302Z namespaces/openshift-dns/pods/node-resolver-wqsgd/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835323264Z namespaces/openshift-dns/pods/node-resolver-wqsgd/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835373285Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835417156Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835471037Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835520258Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83561278Z namespaces/openshift-etcd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835663452Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835724233Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835769754Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835834886Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835897337Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.835987649Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836071591Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836145063Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836208195Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836252096Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836298497Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836328638Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83641344Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836460211Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836501322Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836587404Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836628995Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836671396Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836774119Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836853041Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.836958613Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837037035Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837117387Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83724661Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837407334Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837456745Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837490256Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837542617Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837585988Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83764156Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837687901Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837766363Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837849745Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837933237Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.837979268Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838020679Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83806576Z namespaces/openshift-etcd/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838143612Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838188343Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838202023Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838262615Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838411808Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838612924Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838753457Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.838968622Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839160607Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839203628Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839264899Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.83930031Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839386742Z namespaces/openshift-host-network/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839438684Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839495875Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839538766Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839599377Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839641848Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839729551Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839808822Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839909015Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839960186Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.839995457Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840059459Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840101629Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840181911Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840237073Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840286154Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840365756Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840450728Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840494929Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840595992Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840674714Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840755445Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.840836097Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841086104Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841234047Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84133729Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841394081Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841438352Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841499974Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841545375Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841592726Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841634057Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841694128Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841740189Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841817101Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841916944Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.841977945Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842024986Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842070708Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842101398Z namespaces/openshift-host-network/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84219181Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842237732Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842277453Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies/default-deny.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842328504Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842335854Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842376385Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842526688Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842712303Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.842948009Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843150574Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843338098Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843354828Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84341354Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843456261Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843541643Z namespaces/openshift-image-registry/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843588284Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843663976Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843705347Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843753068Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843795519Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.843920852Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844058996Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844258261Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844304282Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844349503Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844393324Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844439735Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844543568Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844611359Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84465846Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844741552Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844832635Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844843195Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844894126Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.844980148Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.845016639Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.845774587Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84587749Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.846169747Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.846250799Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.846562857Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84669136Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.846904135Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.846967147Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847008087Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847058539Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84710727Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847164591Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847217962Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847299024Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847384546Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847424407Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847476469Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84753921Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847582441Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847683284Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847772166Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847830477Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847879709Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.847976731Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.848010482Z namespaces/openshift-image-registry/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.848126495Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.848184496Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.848219697Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.848269348Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.848421672Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.848604186Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.84874721Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.849031557Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.849234811Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.849246312Z namespaces/openshift-image-registry/pods/image-registry-6c47bcc548-5vsbs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.849286633Z namespaces/openshift-image-registry/pods/image-registry-6c47bcc548-5vsbs/image-registry-6c47bcc548-5vsbs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.849415996Z namespaces/openshift-image-registry/pods/image-registry-6c47bcc548-5vsbs/registry/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.849425616Z namespaces/openshift-image-registry/pods/image-registry-6c47bcc548-5vsbs/registry/registry/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.849433276Z namespaces/openshift-image-registry/pods/image-registry-6c47bcc548-5vsbs/registry/registry/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.849485488Z namespaces/openshift-image-registry/pods/image-registry-6c47bcc548-5vsbs/registry/registry/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.851553438Z namespaces/openshift-image-registry/pods/image-registry-6c47bcc548-5vsbs/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85162492Z namespaces/openshift-image-registry/pods/image-registry-6c47bcc548-5vsbs/registry/registry/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.851673011Z namespaces/openshift-image-registry/pods/node-ca-7cjdl/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.851713702Z namespaces/openshift-image-registry/pods/node-ca-7cjdl/node-ca-7cjdl.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.851801744Z namespaces/openshift-image-registry/pods/node-ca-7cjdl/node-ca/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.851812804Z namespaces/openshift-image-registry/pods/node-ca-7cjdl/node-ca/node-ca/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.851821424Z namespaces/openshift-image-registry/pods/node-ca-7cjdl/node-ca/node-ca/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.851876656Z namespaces/openshift-image-registry/pods/node-ca-7cjdl/node-ca/node-ca/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85205963Z namespaces/openshift-image-registry/pods/node-ca-7cjdl/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852128992Z namespaces/openshift-image-registry/pods/node-ca-7cjdl/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852181683Z namespaces/openshift-image-registry/pods/node-ca-dkmz4/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852214064Z namespaces/openshift-image-registry/pods/node-ca-dkmz4/node-ca-dkmz4.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852310906Z namespaces/openshift-image-registry/pods/node-ca-dkmz4/node-ca/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852322057Z namespaces/openshift-image-registry/pods/node-ca-dkmz4/node-ca/node-ca/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852326997Z namespaces/openshift-image-registry/pods/node-ca-dkmz4/node-ca/node-ca/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852360177Z namespaces/openshift-image-registry/pods/node-ca-dkmz4/node-ca/node-ca/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852540362Z namespaces/openshift-image-registry/pods/node-ca-dkmz4/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852613964Z namespaces/openshift-image-registry/pods/node-ca-dkmz4/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852663635Z namespaces/openshift-image-registry/pods/node-ca-kx7l5/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852703996Z namespaces/openshift-image-registry/pods/node-ca-kx7l5/node-ca-kx7l5.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852773057Z namespaces/openshift-image-registry/pods/node-ca-kx7l5/node-ca/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852783298Z namespaces/openshift-image-registry/pods/node-ca-kx7l5/node-ca/node-ca/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852787438Z namespaces/openshift-image-registry/pods/node-ca-kx7l5/node-ca/node-ca/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.852836299Z namespaces/openshift-image-registry/pods/node-ca-kx7l5/node-ca/node-ca/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853029614Z namespaces/openshift-image-registry/pods/node-ca-kx7l5/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853098335Z namespaces/openshift-image-registry/pods/node-ca-kx7l5/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853153877Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853192388Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853249999Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85328517Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853338531Z namespaces/openshift-infra/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853346701Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853353022Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853400993Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853559616Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853737051Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.853883815Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85411748Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.854327365Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.854370556Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.854655333Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.854695204Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.854758376Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.854781746Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85495158Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855033492Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855113594Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855167615Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855189106Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855246228Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855289538Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855376991Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855424332Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855471283Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855557425Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855634067Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855683558Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855786651Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.855880673Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856054127Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856131829Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856306363Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856422886Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85658738Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856650212Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856697903Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856750504Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856799475Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856856287Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.856938679Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857028941Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857100142Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857146854Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857187505Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857234636Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857272967Z namespaces/openshift-ingress-canary/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857358049Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85741092Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85742665Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857485162Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857639496Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85782805Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.857988464Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858183299Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858505807Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858526337Z namespaces/openshift-ingress-canary/pods/ingress-canary-gt7k2/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858571858Z namespaces/openshift-ingress-canary/pods/ingress-canary-gt7k2/ingress-canary-gt7k2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85865602Z namespaces/openshift-ingress-canary/pods/ingress-canary-gt7k2/serve-healthcheck-canary/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858667731Z namespaces/openshift-ingress-canary/pods/ingress-canary-gt7k2/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858675831Z namespaces/openshift-ingress-canary/pods/ingress-canary-gt7k2/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858717462Z namespaces/openshift-ingress-canary/pods/ingress-canary-gt7k2/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858827135Z namespaces/openshift-ingress-canary/pods/ingress-canary-gt7k2/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858930607Z namespaces/openshift-ingress-canary/pods/ingress-canary-gt7k2/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858951947Z namespaces/openshift-ingress-canary/pods/ingress-canary-hrpz9/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.858993348Z namespaces/openshift-ingress-canary/pods/ingress-canary-hrpz9/ingress-canary-hrpz9.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85907207Z namespaces/openshift-ingress-canary/pods/ingress-canary-hrpz9/serve-healthcheck-canary/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859080151Z namespaces/openshift-ingress-canary/pods/ingress-canary-hrpz9/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859084671Z namespaces/openshift-ingress-canary/pods/ingress-canary-hrpz9/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859119822Z namespaces/openshift-ingress-canary/pods/ingress-canary-hrpz9/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859216494Z namespaces/openshift-ingress-canary/pods/ingress-canary-hrpz9/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859273615Z namespaces/openshift-ingress-canary/pods/ingress-canary-hrpz9/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859331057Z namespaces/openshift-ingress-canary/pods/ingress-canary-rvz8k/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859372888Z namespaces/openshift-ingress-canary/pods/ingress-canary-rvz8k/ingress-canary-rvz8k.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859447819Z namespaces/openshift-ingress-canary/pods/ingress-canary-rvz8k/serve-healthcheck-canary/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85945615Z namespaces/openshift-ingress-canary/pods/ingress-canary-rvz8k/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.85946015Z namespaces/openshift-ingress-canary/pods/ingress-canary-rvz8k/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859490031Z namespaces/openshift-ingress-canary/pods/ingress-canary-rvz8k/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859584683Z namespaces/openshift-ingress-canary/pods/ingress-canary-rvz8k/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859653844Z namespaces/openshift-ingress-canary/pods/ingress-canary-rvz8k/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859688325Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859728877Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859778508Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859815179Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859925311Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.859975172Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860052744Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860108736Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860138966Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860188028Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.86027271Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860350172Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860430234Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860476155Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860515956Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860581327Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860618158Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.86069486Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860741751Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860782202Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860879224Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860951926Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.860996317Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.861754656Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.861835448Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.861979361Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862075914Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862158556Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862296849Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862490344Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862660108Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862694889Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.86276682Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862914664Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862936244Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862940564Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.862976815Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863055267Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863104138Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.86318458Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863265972Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863322194Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863375225Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863459127Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863505878Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8635992Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863652912Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863709133Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863725794Z namespaces/openshift-ingress-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863808225Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863907578Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863918628Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.863965659Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.864055831Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.864065722Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.864118263Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.864274227Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.864453561Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.864590255Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.86479198Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865026015Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865044636Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865106917Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865150298Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865264091Z namespaces/openshift-ingress/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865307432Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.86565131Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865698422Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865757313Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865800894Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.865911867Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866143452Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866377628Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866421659Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.86645834Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866562163Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866601653Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers/istiod-openshift-gateway.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866695286Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers/openshift-ai-inference-openshift-default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866765297Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866795238Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866903671Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866948252Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.866997663Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.867078415Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.867129986Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.867141577Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.867215788Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.867303541Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.867384352Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.867498805Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.867544236Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.868029178Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.868166511Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.86935814Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.869439222Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.87014283Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870272773Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870479958Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8705858Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870608031Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870714293Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870724153Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870760554Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870833036Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870877437Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.870952219Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.87099418Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871081042Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871159024Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871221796Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871267887Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871325258Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871375629Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.87142289Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871461502Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871520703Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871562624Z namespaces/openshift-ingress/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871648016Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871708597Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871723418Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871760839Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.871942933Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872115167Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872257791Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872463386Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872672081Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872683011Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872703302Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/istiod-openshift-gateway-7ddd9cfdc8-wr97s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872813484Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872823755Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872832695Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.872992999Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885372889Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885431471Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-wr97s/discovery/discovery/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885490642Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885511973Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/openshift-ai-inference-openshift-default-86b669b864-2c7vh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885651816Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885661176Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885673917Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885715388Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885835421Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885925893Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-2c7vh/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.885973884Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886021155Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/openshift-ai-inference-openshift-default-86b669b864-5cx6f.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886120278Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886131338Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886141108Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886173909Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886302142Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886369314Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-5cx6f/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886417945Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886458106Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/openshift-ai-inference-openshift-default-86b669b864-8g5tm.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886578549Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886586519Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886597119Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.88663774Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886753123Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886821355Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-8g5tm/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886884056Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.886935587Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/openshift-ai-inference-openshift-default-86b669b864-bv98j.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.88703114Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.88703917Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.88704602Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887094871Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887213944Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887280616Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-bv98j/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887318907Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887352998Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/openshift-ai-inference-openshift-default-86b669b864-njf8w.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.88746586Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887478941Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887485381Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887513541Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887703636Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-njf8w/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887802088Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887834969Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/openshift-ai-inference-openshift-default-86b669b864-nn2ft.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887971413Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887983213Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.887991233Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888033804Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.88825299Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-nn2ft/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888371542Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888430544Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/openshift-ai-inference-openshift-default-86b669b864-rd99c.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888531236Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888542306Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888552507Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888603058Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888820413Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-rd99c/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.888985667Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889039619Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/openshift-ai-inference-openshift-default-86b669b864-t2dz9.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889149401Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889160681Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889165622Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889221913Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889361906Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889437768Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-t2dz9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889476839Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.88950739Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/openshift-ai-inference-openshift-default-86b669b864-wvd5p.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889600802Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889609312Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889625253Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889680024Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889796087Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889880169Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-wvd5p/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.88990633Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.889956441Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/openshift-ai-inference-openshift-default-86b669b864-zctx5.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890074964Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890087174Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890099734Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890137965Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890261828Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.89032431Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-86b669b864-zctx5/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890377771Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890418182Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router-default-6668ff59b7-vglrn.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890539435Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890546645Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890554946Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890599527Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890714049Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890786511Z namespaces/openshift-ingress/pods/router-default-6668ff59b7-vglrn/router/router/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890833472Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890879523Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890946315Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.890983626Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891082808Z namespaces/openshift-insights/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8911314Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891255863Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891290153Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891364515Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891401416Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891524669Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891638482Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891742934Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891783865Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891831766Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.891930049Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.89196501Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.892018541Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.892057812Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.892141724Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.892212776Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.892256597Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.892336599Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8923989Z namespaces/openshift-insights/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.892440751Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.893347153Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.893458026Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.893790744Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.893942208Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894266466Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894352508Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894523652Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894593604Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894639545Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894705266Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894743747Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.89483601Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894915361Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.894960373Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895021044Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895049565Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895118556Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895162517Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895218299Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.89525757Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895341872Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895420324Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895469085Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895510696Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895568137Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895604278Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.89567086Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895710531Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895766162Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895796163Z namespaces/openshift-insights/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895906956Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895959047Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895966777Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.895994748Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.896187972Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.896409708Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.896572342Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.896765826Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.896995722Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897006392Z namespaces/openshift-insights/pods/insights-operator-7c4986d449-xkcch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897031633Z namespaces/openshift-insights/pods/insights-operator-7c4986d449-xkcch/insights-operator-7c4986d449-xkcch.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897134525Z namespaces/openshift-insights/pods/insights-operator-7c4986d449-xkcch/insights-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897142646Z namespaces/openshift-insights/pods/insights-operator-7c4986d449-xkcch/insights-operator/insights-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897148996Z namespaces/openshift-insights/pods/insights-operator-7c4986d449-xkcch/insights-operator/insights-operator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897184296Z namespaces/openshift-insights/pods/insights-operator-7c4986d449-xkcch/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897395812Z namespaces/openshift-insights/pods/insights-operator-7c4986d449-xkcch/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897511795Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897554925Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/insights-runtime-extractor-5rttv.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897643708Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897650748Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/exporter/exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897659398Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/exporter/exporter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897703249Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/exporter/exporter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897797991Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897873403Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/exporter/exporter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897904664Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/extractor/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897913054Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/extractor/extractor/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897920284Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/extractor/extractor/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.897965866Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/extractor/extractor/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898068948Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.89812919Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/extractor/extractor/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.89816552Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8981725Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898184891Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898219462Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898311084Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898379125Z namespaces/openshift-insights/pods/insights-runtime-extractor-5rttv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898433257Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898460218Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/insights-runtime-extractor-6mwl7.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.89857003Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8985776Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/exporter/exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898581501Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/exporter/exporter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898612191Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/exporter/exporter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898705373Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898768685Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/exporter/exporter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898811916Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/extractor/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898823496Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/extractor/extractor/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898828536Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/extractor/extractor/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.898850607Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/extractor/extractor/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.8989616Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899035492Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/extractor/extractor/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899064872Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899071762Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899075793Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899113614Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899203506Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899270967Z namespaces/openshift-insights/pods/insights-runtime-extractor-6mwl7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899316958Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899345969Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/insights-runtime-extractor-6w8dk.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899438691Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899445562Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/exporter/exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899449442Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/exporter/exporter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899479922Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/exporter/exporter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899575975Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899643066Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/exporter/exporter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899673307Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/extractor/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899684537Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/extractor/extractor/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899691948Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/extractor/extractor/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899724828Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/extractor/extractor/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899822381Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899906283Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/extractor/extractor/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899937394Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899946604Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899950604Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.899996515Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900082307Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900147959Z namespaces/openshift-insights/pods/insights-runtime-extractor-6w8dk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900179629Z namespaces/openshift-insights/pods/periodic-gathering-mxxhw-rrcns/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.90022368Z namespaces/openshift-insights/pods/periodic-gathering-mxxhw-rrcns/periodic-gathering-mxxhw-rrcns.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900304702Z namespaces/openshift-insights/pods/periodic-gathering-mxxhw-rrcns/insights-gathering/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900313363Z namespaces/openshift-insights/pods/periodic-gathering-mxxhw-rrcns/insights-gathering/insights-gathering/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900318263Z namespaces/openshift-insights/pods/periodic-gathering-mxxhw-rrcns/insights-gathering/insights-gathering/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900341833Z namespaces/openshift-insights/pods/periodic-gathering-mxxhw-rrcns/insights-gathering/insights-gathering/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900917737Z namespaces/openshift-insights/pods/periodic-gathering-mxxhw-rrcns/insights-gathering/insights-gathering/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.900990209Z namespaces/openshift-insights/pods/periodic-gathering-mxxhw-rrcns/insights-gathering/insights-gathering/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.90102511Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901059521Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901113392Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901147733Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901206864Z namespaces/openshift-keda/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901213864Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901217774Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901255535Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901351948Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901396599Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901403749Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.90144204Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901498091Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901563333Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901647645Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901739797Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901793239Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.901800299Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9018503Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.902022114Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.902212319Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-4.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.902412334Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.902633219Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.902777112Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.902984357Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903023418Z namespaces/openshift-keda/operators.coreos.com/installplans/install-v499b.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903135761Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903171032Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-4.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903231123Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903271715Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903331416Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903362197Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903444039Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903476989Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903546201Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903575482Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903638553Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903672024Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903756416Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903835278Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903950771Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.903986802Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904041093Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904092264Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904122285Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904211657Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904261579Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904304909Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904385471Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904429432Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904463073Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904578866Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904654418Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.90473431Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904810672Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.904938785Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905063848Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905247162Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905296673Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905337285Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905398446Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905420096Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905472368Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905517779Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905593981Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905672003Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905718864Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905754205Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905812786Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905858897Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.90596663Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905988541Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.905998451Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.906044972Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.906199326Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.90637611Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.906511153Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.906734808Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.906943434Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.906960694Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907033726Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907053436Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907131378Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907176659Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907243291Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907280132Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907338153Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907377894Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907461446Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907536108Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.90761769Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907666501Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907703922Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907751063Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907793064Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907875416Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907934048Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.907962878Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908050491Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908106952Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908144603Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908239725Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908317927Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908396819Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908476581Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908553023Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908676556Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908832739Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908902691Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908952693Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.908994534Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909034494Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909085406Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909129837Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909211619Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909284581Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909324631Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909382273Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909437764Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909476355Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909576197Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909634139Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9096867Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909733151Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909761182Z namespaces/openshift-kube-apiserver/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909852044Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909909456Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909928686Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.909974847Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.910121091Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.910381367Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.910534771Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.910733486Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.910937231Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.910973042Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911029013Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911079054Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911141456Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911174967Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911236218Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911279109Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911340051Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911369921Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911460134Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911536655Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911632018Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911667008Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911713049Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911764761Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911811902Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911912154Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911955266Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.911997996Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.912079009Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.91213702Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.912184501Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.912290883Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.912374756Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.912453178Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.91253438Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.912620542Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.912747335Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913004121Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913061222Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913110823Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913149114Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913193605Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913243787Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913294038Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.91337184Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913451012Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913494413Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913537834Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913601615Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913626786Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913692758Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913727538Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913827721Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913879062Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913896182Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.913955744Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.914100978Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.914278822Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.914414005Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.914636701Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.914812815Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.914827855Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.914917547Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.914937208Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915055851Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915092572Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915167544Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915201434Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915268836Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915301707Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915386729Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915464501Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915543593Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915580154Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915628505Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915691866Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915736607Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915816319Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915875451Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.915942452Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.916020614Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.916067885Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.916078336Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.916129377Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.916203369Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.91625195Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.916351972Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.916429784Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.916967207Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917049079Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917181162Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917340816Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.91750756Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917574812Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917627293Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917680565Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917737136Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917789337Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917829598Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.917942231Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918022153Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918078954Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918134456Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918184147Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918234428Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.91831729Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918355221Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918383012Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918437523Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918602177Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918785812Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.918946745Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.91915733Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919343435Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919394686Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919452718Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919485718Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919574041Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919618982Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919707334Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919750335Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919794956Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.919841697Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.91994551Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920027702Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920110984Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920156815Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920205476Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920243257Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920286468Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920394271Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920444872Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920501363Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920577205Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920639886Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920718618Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920824641Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.920983925Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921057456Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921144969Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921230931Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921357014Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921524098Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921567059Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92162293Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921673831Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921712052Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921762664Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921795345Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921892437Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.921984529Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92202403Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922069501Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922113372Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922147353Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922235485Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922287677Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922300247Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922325727Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922487941Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922752378Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.922888771Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923107886Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923285761Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923303391Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923370773Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923407884Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923477705Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923514956Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923574098Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923607809Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92367528Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923701931Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923792403Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923883465Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.923975107Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924016919Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92406166Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924108341Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924159632Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924242044Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924280485Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924311956Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924400058Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924459139Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92449243Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924596382Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924683975Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924764817Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924847869Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.924957671Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925073204Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925229798Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925280309Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92531499Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925375972Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925400842Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925453673Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925487384Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925578406Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925653358Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925702709Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92573415Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925789562Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925829012Z namespaces/openshift-kube-scheduler/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925933185Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925966716Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.925973666Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.926040088Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.926176941Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.926355145Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.926490659Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.926703264Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.926902429Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92694647Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.926998251Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927036982Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927098033Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927141664Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927194106Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927233017Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927288598Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927329539Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927414461Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927528504Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927637347Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927679288Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927722018Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92777317Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92780828Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927930214Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.927982825Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928024186Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928104748Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928152569Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928160499Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92820281Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928260151Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928309223Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928407925Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928493957Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928674452Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.928750863Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92901487Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929139203Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929336118Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929392499Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92944427Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929480841Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929532613Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929587804Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929615115Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929707627Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929785999Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.92982708Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929877761Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929934192Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.929975623Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.930058325Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.930091166Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.930099846Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.930157618Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.930324242Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.930506346Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.930644229Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.930842544Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93105019Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93106055Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93109796Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931254834Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b/kube-storage-version-migrator-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931267275Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931276285Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931289525Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931412068Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931540611Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931570792Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931634244Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931665754Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931744516Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931777507Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931849639Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93189715Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931970792Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.931989212Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932085284Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932193357Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932275649Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9323226Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932372582Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932434683Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932468054Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932554176Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932598487Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932633798Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93271439Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932769631Z namespaces/openshift-kube-storage-version-migrator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932781732Z namespaces/openshift-kube-storage-version-migrator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932794342Z namespaces/openshift-kube-storage-version-migrator/coordination.k8s.io/leases/storage-version-migration-migrator-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932875804Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.932900314Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933027067Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933106099Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933230032Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933306944Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933439488Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9335646Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933730935Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933768625Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933816997Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933879068Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.933922599Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93396903Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934014791Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934102753Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934178026Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934228537Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934260817Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934314919Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93435875Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934435172Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934487853Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934496923Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934547915Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934716479Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.934919113Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935120878Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935316343Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935500607Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935513718Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-ds7bd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935560399Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-ds7bd/migrator-66df5d496b-ds7bd.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935637311Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-ds7bd/migrator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935648471Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-ds7bd/migrator/migrator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935653741Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-ds7bd/migrator/migrator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935673452Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-ds7bd/migrator/migrator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935772534Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-ds7bd/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935848656Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-ds7bd/migrator/migrator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935897707Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-vfc6j/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.935938638Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-vfc6j/migrator-66df5d496b-vfc6j.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93601094Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-vfc6j/migrator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936036731Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-vfc6j/migrator/migrator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936043391Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-vfc6j/migrator/migrator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936062681Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-vfc6j/migrator/migrator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936164624Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-vfc6j/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936232785Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-66df5d496b-vfc6j/migrator/migrator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936268756Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936301717Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936360608Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936399319Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936476561Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936527292Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936583314Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936619775Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936672656Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936716787Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.936793039Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937006164Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937136127Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937181758Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937226609Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93725951Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937306982Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937388684Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937433325Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937478776Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937558217Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937599589Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937606739Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93764948Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937729722Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937785753Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937832624Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.937954357Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.938037789Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.938363487Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.938447999Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.938649684Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.938808458Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939012443Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939075874Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939122446Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939190417Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939222618Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93928851Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93930719Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939410202Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939507855Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939559466Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939598187Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939662939Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.93971054Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939787072Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939837163Z namespaces/openshift-lws-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939949055Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939989736Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.939998167Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.940056218Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.940209232Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.940393906Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94053104Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.940744365Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94094983Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941135664Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941175576Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-sm89h.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941257727Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941288908Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94137134Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941401621Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941468352Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941500293Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941576955Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941587086Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-8t7zf/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941630967Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-8t7zf/lws-controller-manager-6667976c75-8t7zf.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941718869Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-8t7zf/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941727899Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-8t7zf/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941732729Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-8t7zf/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94177354Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-8t7zf/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.941976745Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-8t7zf/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942099598Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-8t7zf/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942106478Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-tdwkq/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942134949Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-tdwkq/lws-controller-manager-6667976c75-tdwkq.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942221211Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-tdwkq/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942228531Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-tdwkq/manager/manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942232411Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-tdwkq/manager/manager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942276912Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-tdwkq/manager/manager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942377485Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-tdwkq/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942456867Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6667976c75-tdwkq/manager/manager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942490817Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-8465994787-n5xg6/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942552489Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-8465994787-n5xg6/openshift-lws-operator-8465994787-n5xg6.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942654071Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-8465994787-n5xg6/openshift-lws-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942661172Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-8465994787-n5xg6/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942671042Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-8465994787-n5xg6/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942718783Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-8465994787-n5xg6/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942908978Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-8465994787-n5xg6/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.942981699Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-8465994787-n5xg6/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94301411Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943072331Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943113952Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943162254Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943212645Z namespaces/openshift-machine-api/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943221715Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943225695Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943282827Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94343049Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943616375Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943753238Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.943984454Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944173268Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944182849Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944187829Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944228859Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944347513Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944424474Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944435265Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944479946Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94464625Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.944829364Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94504389Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945248834Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945438869Z namespaces/openshift-marketplace/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945446269Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945450269Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94547628Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945543322Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945557302Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945593333Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945684635Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945777907Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.945816728Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94587653Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946038443Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946235068Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946371462Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946577597Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946769821Z namespaces/openshift-monitoring/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946809892Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946874154Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946925305Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.946973396Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.947017517Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.947156641Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.947596131Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948045762Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948284768Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948321099Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94837668Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948422872Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948508523Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948549235Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948597096Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948706528Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.94875855Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.948809201Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.951553128Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.951692511Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.952801498Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.95288008Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.953969866Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.954096939Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.954575261Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.954750695Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.954791576Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.95495082Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.95497266Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955049513Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955068713Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955165335Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955257938Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955303009Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9553509Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955518364Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955558245Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955642547Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955681968Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.95579236Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955839382Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.955952615Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.956117548Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.956206431Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.956477437Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.95660936Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.956733094Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.956826606Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.956950469Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957041001Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957100172Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957153464Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957234906Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957317968Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics-minimal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9574298Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957512252Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet-minimal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957685867Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957794479Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.957920742Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter-minimal.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958018325Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958112727Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958214299Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958298961Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958422695Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958530777Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.95863739Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958674021Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958727642Z namespaces/openshift-monitoring/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958804954Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958913806Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958929197Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.958957968Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.959138552Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.959347087Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.95948813Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.959712006Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.959969712Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960014953Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960100475Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960111376Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960175657Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960359491Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960368052Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960372952Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960408063Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960507455Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960573047Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960616428Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960625108Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960629208Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960666089Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960767172Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960834193Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960875164Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960951866Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.960960486Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961002037Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96109526Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961164901Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961196902Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961207652Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961219543Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961248893Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961339465Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961405837Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961453808Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961463629Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961467718Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961503029Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961588541Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961656373Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961697054Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961706584Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961716964Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961747995Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961838747Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96193216Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961969121Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961979081Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.961985021Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.962006942Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.962092764Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.962162005Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.962197706Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-884576c57-4vs7k/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.962239707Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-884576c57-4vs7k/cluster-monitoring-operator-884576c57-4vs7k.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96233129Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-884576c57-4vs7k/cluster-monitoring-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96234242Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-884576c57-4vs7k/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96234701Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-884576c57-4vs7k/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.962383361Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-884576c57-4vs7k/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963138749Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-884576c57-4vs7k/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963200311Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-884576c57-4vs7k/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963246962Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963289093Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-state-metrics-5dc8d7d89b-p4pkz.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963387455Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-main/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963397855Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963403785Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963477537Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9635762Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963643991Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963682742Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-self/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963706093Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963719033Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963747834Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963837996Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963937509Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963968249Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-state-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.963972679Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96399314Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964052721Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964326408Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96440893Z namespaces/openshift-monitoring/pods/kube-state-metrics-5dc8d7d89b-p4pkz/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96442819Z namespaces/openshift-monitoring/pods/metrics-server-5f5784d99-gnbws/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964471302Z namespaces/openshift-monitoring/pods/metrics-server-5f5784d99-gnbws/metrics-server-5f5784d99-gnbws.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964568354Z namespaces/openshift-monitoring/pods/metrics-server-5f5784d99-gnbws/metrics-server/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964577024Z namespaces/openshift-monitoring/pods/metrics-server-5f5784d99-gnbws/metrics-server/metrics-server/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964584974Z namespaces/openshift-monitoring/pods/metrics-server-5f5784d99-gnbws/metrics-server/metrics-server/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964623025Z namespaces/openshift-monitoring/pods/metrics-server-5f5784d99-gnbws/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964723618Z namespaces/openshift-monitoring/pods/metrics-server-5f5784d99-gnbws/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964790739Z namespaces/openshift-monitoring/pods/metrics-server-5f5784d99-gnbws/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964841611Z namespaces/openshift-monitoring/pods/monitoring-plugin-77f4f76594-249ck/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964884592Z namespaces/openshift-monitoring/pods/monitoring-plugin-77f4f76594-249ck/monitoring-plugin-77f4f76594-249ck.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.964988154Z namespaces/openshift-monitoring/pods/monitoring-plugin-77f4f76594-249ck/monitoring-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965001474Z namespaces/openshift-monitoring/pods/monitoring-plugin-77f4f76594-249ck/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965008424Z namespaces/openshift-monitoring/pods/monitoring-plugin-77f4f76594-249ck/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965018075Z namespaces/openshift-monitoring/pods/monitoring-plugin-77f4f76594-249ck/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965130458Z namespaces/openshift-monitoring/pods/monitoring-plugin-77f4f76594-249ck/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965193889Z namespaces/openshift-monitoring/pods/monitoring-plugin-77f4f76594-249ck/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96523972Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965267431Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/node-exporter-2j5mw.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965378864Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/init-textfile/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965386884Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/init-textfile/init-textfile/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965391024Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/init-textfile/init-textfile/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965423205Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965494556Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965562978Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965589759Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965594889Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965604329Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96564299Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965732612Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965804264Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965844935Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/node-exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965853285Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/node-exporter/node-exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965857525Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/node-exporter/node-exporter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.965909257Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966019989Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966089141Z namespaces/openshift-monitoring/pods/node-exporter-2j5mw/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966143012Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966190333Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/node-exporter-5h8g5.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966290846Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/init-textfile/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966299256Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/init-textfile/init-textfile/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966309146Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/init-textfile/init-textfile/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966343827Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966420249Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966489091Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966560132Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966567592Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966571753Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966617974Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966701536Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966770257Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966808298Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/node-exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966814568Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/node-exporter/node-exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.966818568Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/node-exporter/node-exporter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96687947Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967000643Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967068454Z namespaces/openshift-monitoring/pods/node-exporter-5h8g5/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967107646Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967145806Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/node-exporter-jxc4l.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967245849Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/init-textfile/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967252969Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/init-textfile/init-textfile/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967256939Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/init-textfile/init-textfile/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96728977Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967366602Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967434043Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967466814Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967473654Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967479704Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967517656Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967610048Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967680149Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96772575Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/node-exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967735791Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/node-exporter/node-exporter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967739791Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/node-exporter/node-exporter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967766002Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967883314Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967955356Z namespaces/openshift-monitoring/pods/node-exporter-jxc4l/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.967983687Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968029358Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/openshift-state-metrics-7bd7489957-l6wf8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968165991Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-main/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968173091Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968177041Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968220233Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968312445Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968379757Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968428168Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-self/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968438418Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968444768Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968469659Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968561891Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968625483Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968656353Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/openshift-state-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968666003Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968683104Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968713585Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968839948Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968954631Z namespaces/openshift-monitoring/pods/openshift-state-metrics-7bd7489957-l6wf8/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.968995832Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969068093Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969251888Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969259028Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969263048Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969303659Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969406131Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969474103Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969509104Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969516054Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969520014Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969560725Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969653088Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969725069Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96974181Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96974926Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.96975641Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.969804571Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970029106Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970096948Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970130299Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970138139Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97014762Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97017535Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970277243Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970343934Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970377845Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970385385Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970390215Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970428716Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970527379Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9706062Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970624731Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970632261Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970641851Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.970701603Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971267487Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971332608Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971372189Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971378689Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971382739Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97142081Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971518283Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971592604Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971635776Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971691037Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/prometheus-operator-9cd89b74c-9dp62.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971777599Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971785729Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971789709Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97182861Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.971940053Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972009225Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972044715Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/prometheus-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972051646Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972055846Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972087447Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972325072Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972391434Z namespaces/openshift-monitoring/pods/prometheus-operator-9cd89b74c-9dp62/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972417704Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-6489d8bf89-wwjmw/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972464376Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-6489d8bf89-wwjmw/prometheus-operator-admission-webhook-6489d8bf89-wwjmw.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972587979Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-6489d8bf89-wwjmw/prometheus-operator-admission-webhook/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972594959Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-6489d8bf89-wwjmw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972599249Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-6489d8bf89-wwjmw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97264168Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-6489d8bf89-wwjmw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972733872Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-6489d8bf89-wwjmw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972800634Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-6489d8bf89-wwjmw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972847305Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.972896456Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/telemeter-client-56c84f9555-d5rqg.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973074111Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973081361Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973085361Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973114652Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973212344Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973278965Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973315056Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/reload/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973321997Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/reload/reload/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973325786Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/reload/reload/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973363038Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/reload/reload/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97345346Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973525131Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/reload/reload/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973557412Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/telemeter-client/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973564342Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/telemeter-client/telemeter-client/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973568323Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973605053Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97385663Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.973959482Z namespaces/openshift-monitoring/pods/telemeter-client-56c84f9555-d5rqg/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974010653Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974083135Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/thanos-querier-6776c6b778-2gtz5.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974214148Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974225408Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974231968Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974245329Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974355751Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974427173Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974451144Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-rules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974455514Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974466494Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974529456Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974620648Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97469261Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974736491Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-web/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974743811Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974747881Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974781222Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974924985Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.974999847Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975018968Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975026588Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975036758Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97509654Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975183302Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975256373Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975284894Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/prom-label-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975293504Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975297554Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975339926Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975431228Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97550053Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97553319Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/thanos-query/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9755399Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/thanos-query/thanos-query/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97554606Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/thanos-query/thanos-query/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975580251Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975696194Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975764066Z namespaces/openshift-monitoring/pods/thanos-querier-6776c6b778-2gtz5/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975798677Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975825497Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.975904279Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97593599Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976030092Z namespaces/openshift-multus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976098814Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976165386Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976198836Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976253458Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97632161Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976523264Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976603806Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976685758Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9767439Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.976804151Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977100588Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977214841Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977278963Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977316183Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977372365Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977406626Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977502118Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977544699Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.97759778Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977690973Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977739414Z namespaces/openshift-multus/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977792325Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.977942399Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.978013831Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.978543193Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.978673476Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.979273171Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.979371264Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.979505757Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.979598719Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.979666601Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.979765173Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.979878706Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.979963448Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98006Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980095831Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980193673Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980272575Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980356688Z namespaces/openshift-multus/core/serviceaccounts/multus-node-identity.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980438079Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980485851Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980536162Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980576123Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980630824Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980682675Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980733107Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980770138Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980799318Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980904891Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.980986433Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981027694Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981072475Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981124516Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981173237Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981225109Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981265099Z namespaces/openshift-multus/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981342401Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981401653Z namespaces/openshift-multus/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981449164Z namespaces/openshift-multus/networking.k8s.io/networkpolicies/default-deny.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981528656Z namespaces/openshift-multus/networking.k8s.io/networkpolicies/multus-admission-controller.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981611778Z namespaces/openshift-multus/networking.k8s.io/networkpolicies/network-metrics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981673419Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981682Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981719941Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.981962766Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982142051Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982279234Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982471719Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982674374Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982684064Z namespaces/openshift-multus/pods/multus-4bwqs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982727375Z namespaces/openshift-multus/pods/multus-4bwqs/multus-4bwqs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982815627Z namespaces/openshift-multus/pods/multus-4bwqs/kube-multus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982824157Z namespaces/openshift-multus/pods/multus-4bwqs/kube-multus/kube-multus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982830088Z namespaces/openshift-multus/pods/multus-4bwqs/kube-multus/kube-multus/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.982882399Z namespaces/openshift-multus/pods/multus-4bwqs/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.983520684Z namespaces/openshift-multus/pods/multus-4bwqs/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.983587256Z namespaces/openshift-multus/pods/multus-4bwqs/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.983619527Z namespaces/openshift-multus/pods/multus-4gvnm/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.983698699Z namespaces/openshift-multus/pods/multus-4gvnm/multus-4gvnm.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.983791601Z namespaces/openshift-multus/pods/multus-4gvnm/kube-multus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.983804631Z namespaces/openshift-multus/pods/multus-4gvnm/kube-multus/kube-multus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.983814452Z namespaces/openshift-multus/pods/multus-4gvnm/kube-multus/kube-multus/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.983827242Z namespaces/openshift-multus/pods/multus-4gvnm/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98499189Z namespaces/openshift-multus/pods/multus-4gvnm/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985056322Z namespaces/openshift-multus/pods/multus-4gvnm/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985106293Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985150984Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/multus-additional-cni-plugins-64cvm.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985278127Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/bond-cni-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985285187Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985289107Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985322238Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98541257Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985478942Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985510033Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985517183Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/cni-plugins/cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985521243Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985565724Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985657436Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985718468Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985765289Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/egress-router-binary-copy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985772469Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985776719Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98581013Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985928033Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.985993074Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986035625Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986047166Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986053166Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986082976Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986155898Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98621999Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986266111Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/routeoverride-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986277891Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986282911Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986299472Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986389754Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986455445Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986490546Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni-bincopy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986497337Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986501297Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986554558Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98664822Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986706572Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986783444Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986794614Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986801014Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986817434Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.986934337Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987006189Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-64cvm/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98704078Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987077111Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/multus-additional-cni-plugins-gbckc.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987193674Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/bond-cni-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987200864Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987204844Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987242185Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987333177Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987401399Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98744337Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9874557Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/cni-plugins/cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98745997Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987490831Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987582313Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987652125Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987688576Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/egress-router-binary-copy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987695506Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987699596Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987730117Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987821309Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987909681Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987935242Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987945182Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987951112Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.987990793Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988057875Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988132516Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988159107Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/routeoverride-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988183628Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988193528Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988199988Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98828026Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988364482Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988383052Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni-bincopy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988393773Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988403883Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988439134Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988526346Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988594338Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988610878Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988617758Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988625388Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98867688Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988759982Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988827503Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gbckc/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988844934Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.988925706Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/multus-additional-cni-plugins-kl9nq.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989033538Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/bond-cni-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989042088Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989046179Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989073409Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989179182Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989240333Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989257784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989266524Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/cni-plugins/cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989278124Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989318415Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989405537Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989478919Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98951786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/egress-router-binary-copy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9895344Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98954072Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989562211Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989655563Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989722855Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989756786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989766046Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989798267Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989840488Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.98994097Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.989995082Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990025952Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/routeoverride-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990042603Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990046923Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990084214Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990164616Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990228887Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990263948Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni-bincopy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990271008Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990275168Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990312309Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990425312Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990492754Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990533095Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990539675Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990543605Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990588616Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990678928Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.99074468Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kl9nq/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990789271Z namespaces/openshift-multus/pods/multus-kfshl/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990825682Z namespaces/openshift-multus/pods/multus-kfshl/multus-kfshl.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990931024Z namespaces/openshift-multus/pods/multus-kfshl/kube-multus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990950165Z namespaces/openshift-multus/pods/multus-kfshl/kube-multus/kube-multus/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990956395Z namespaces/openshift-multus/pods/multus-kfshl/kube-multus/kube-multus/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.990969365Z namespaces/openshift-multus/pods/multus-kfshl/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992239326Z namespaces/openshift-multus/pods/multus-kfshl/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992307758Z namespaces/openshift-multus/pods/multus-kfshl/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992333919Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.99238243Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/network-metrics-daemon-4pvll.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992466932Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992473752Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992477802Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992517423Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992609925Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992676627Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992704857Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/network-metrics-daemon/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992714158Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992720468Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.992751059Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993029985Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993102827Z namespaces/openshift-multus/pods/network-metrics-daemon-4pvll/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993131198Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993178149Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/network-metrics-daemon-cdmr4.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993256001Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993266091Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993270841Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993295522Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993398204Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993465616Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993503657Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/network-metrics-daemon/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993513007Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993517027Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993562348Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993712372Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993777414Z namespaces/openshift-multus/pods/network-metrics-daemon-cdmr4/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993817195Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993877796Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/network-metrics-daemon-hmgsj.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993957278Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993967398Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.993971478Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994020949Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994121572Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994188033Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994241035Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/network-metrics-daemon/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994251695Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994264945Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994285556Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994488481Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994569273Z namespaces/openshift-multus/pods/network-metrics-daemon-hmgsj/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994604794Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994640095Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994692296Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994701266Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994725166Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994819739Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.99487106Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.994930571Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995014193Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995064195Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995105606Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995156847Z namespaces/openshift-must-gather-dn4ld/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995163637Z namespaces/openshift-must-gather-dn4ld/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995169837Z namespaces/openshift-must-gather-dn4ld/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995222539Z namespaces/openshift-must-gather-dn4ld/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995379233Z namespaces/openshift-must-gather-dn4ld/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995630328Z namespaces/openshift-must-gather-dn4ld/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.995795723Z namespaces/openshift-must-gather-dn4ld/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996011548Z namespaces/openshift-must-gather-dn4ld/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996184052Z namespaces/openshift-network-console/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996227703Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996287364Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996320535Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996371606Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996404097Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996494809Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996616372Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996729755Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996749966Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996805167Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996885379Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.99693735Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.996990801Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997036523Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997122125Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997148685Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997197466Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997280739Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.99732641Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997367131Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997503354Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997588996Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997697449Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.997827762Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998045297Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998132199Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998241512Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998304624Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998335614Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998400386Z namespaces/openshift-network-console/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998428967Z namespaces/openshift-network-console/core/serviceaccounts/networking-console-plugin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998477828Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998543999Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.9985865Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998637102Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998699163Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998719014Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998764225Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998814006Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.998907898Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.99899383Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999042771Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999084903Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999132084Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999168994Z namespaces/openshift-network-console/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999258117Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999309848Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999344069Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies/default-deny.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999429831Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies/networking-console-plugin.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999473742Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999480652Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999536114Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999694857Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:41.999885752Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000052136Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00023942Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000419245Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000429675Z namespaces/openshift-network-console/pods/networking-console-plugin-86c478bb88-grpzl/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000451726Z namespaces/openshift-network-console/pods/networking-console-plugin-86c478bb88-grpzl/networking-console-plugin-86c478bb88-grpzl.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000541998Z namespaces/openshift-network-console/pods/networking-console-plugin-86c478bb88-grpzl/networking-console-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000551228Z namespaces/openshift-network-console/pods/networking-console-plugin-86c478bb88-grpzl/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000557998Z namespaces/openshift-network-console/pods/networking-console-plugin-86c478bb88-grpzl/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000590809Z namespaces/openshift-network-console/pods/networking-console-plugin-86c478bb88-grpzl/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000689802Z namespaces/openshift-network-console/pods/networking-console-plugin-86c478bb88-grpzl/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000760513Z namespaces/openshift-network-console/pods/networking-console-plugin-86c478bb88-grpzl/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000778493Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000835215Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000887526Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.000986039Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0010272Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001077881Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001129702Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001179623Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001220244Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001264585Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001378288Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00146927Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001569103Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001599044Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001645485Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001715956Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001762868Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001821379Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00187615Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001924181Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.001954442Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002045784Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002228529Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002247349Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002343892Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002374443Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002418833Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002532826Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002622909Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002825793Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.002925946Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003131231Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003216353Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003328646Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003385377Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003425868Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-check-target.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00350973Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003559401Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003604172Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003685724Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003737176Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003784037Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003839358Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003898809Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003946581Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.003976491Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004063384Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004143055Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004185287Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004230868Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004283959Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004329Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004377041Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004418192Z namespaces/openshift-network-diagnostics/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004497994Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004554515Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004598646Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies/default-deny.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004678978Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies/network-check-source.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00476165Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies/network-check-target.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004810702Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004818512Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.004852732Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005024417Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005214271Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005345724Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005544099Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005725224Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005732964Z namespaces/openshift-network-diagnostics/pods/network-check-source-6f78d59f99-5s5g6/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005772615Z namespaces/openshift-network-diagnostics/pods/network-check-source-6f78d59f99-5s5g6/network-check-source-6f78d59f99-5s5g6.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005847027Z namespaces/openshift-network-diagnostics/pods/network-check-source-6f78d59f99-5s5g6/check-endpoints/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005854117Z namespaces/openshift-network-diagnostics/pods/network-check-source-6f78d59f99-5s5g6/check-endpoints/check-endpoints/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005858027Z namespaces/openshift-network-diagnostics/pods/network-check-source-6f78d59f99-5s5g6/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.005920558Z namespaces/openshift-network-diagnostics/pods/network-check-source-6f78d59f99-5s5g6/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006018431Z namespaces/openshift-network-diagnostics/pods/network-check-source-6f78d59f99-5s5g6/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006089923Z namespaces/openshift-network-diagnostics/pods/network-check-source-6f78d59f99-5s5g6/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006116813Z namespaces/openshift-network-diagnostics/pods/network-check-target-27nh7/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006166695Z namespaces/openshift-network-diagnostics/pods/network-check-target-27nh7/network-check-target-27nh7.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006255027Z namespaces/openshift-network-diagnostics/pods/network-check-target-27nh7/network-check-target-container/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006263197Z namespaces/openshift-network-diagnostics/pods/network-check-target-27nh7/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006267647Z namespaces/openshift-network-diagnostics/pods/network-check-target-27nh7/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006301798Z namespaces/openshift-network-diagnostics/pods/network-check-target-27nh7/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00639668Z namespaces/openshift-network-diagnostics/pods/network-check-target-27nh7/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006462892Z namespaces/openshift-network-diagnostics/pods/network-check-target-27nh7/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006498133Z namespaces/openshift-network-diagnostics/pods/network-check-target-gtwvr/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006545444Z namespaces/openshift-network-diagnostics/pods/network-check-target-gtwvr/network-check-target-gtwvr.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006621736Z namespaces/openshift-network-diagnostics/pods/network-check-target-gtwvr/network-check-target-container/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006628716Z namespaces/openshift-network-diagnostics/pods/network-check-target-gtwvr/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006632736Z namespaces/openshift-network-diagnostics/pods/network-check-target-gtwvr/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006673167Z namespaces/openshift-network-diagnostics/pods/network-check-target-gtwvr/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006761869Z namespaces/openshift-network-diagnostics/pods/network-check-target-gtwvr/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006831031Z namespaces/openshift-network-diagnostics/pods/network-check-target-gtwvr/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006876042Z namespaces/openshift-network-diagnostics/pods/network-check-target-tgn5k/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006922073Z namespaces/openshift-network-diagnostics/pods/network-check-target-tgn5k/network-check-target-tgn5k.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.006998595Z namespaces/openshift-network-diagnostics/pods/network-check-target-tgn5k/network-check-target-container/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007007135Z namespaces/openshift-network-diagnostics/pods/network-check-target-tgn5k/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007011395Z namespaces/openshift-network-diagnostics/pods/network-check-target-tgn5k/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007049676Z namespaces/openshift-network-diagnostics/pods/network-check-target-tgn5k/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007138808Z namespaces/openshift-network-diagnostics/pods/network-check-target-tgn5k/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00720823Z namespaces/openshift-network-diagnostics/pods/network-check-target-tgn5k/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007268401Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007309202Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007363874Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007371374Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007403855Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007489787Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007553898Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007570799Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007662801Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007714912Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007732843Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007803554Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.007832085Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008075231Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008123722Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008153633Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008203884Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008287666Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008373448Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00845562Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008506281Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008552653Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008597624Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008637844Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008714266Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008781958Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008799248Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.008890871Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009035314Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009048645Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009063875Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009135247Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009151097Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.00925745Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009335761Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009414394Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009495836Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009574357Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009725121Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009833174Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009902165Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.009939026Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01011126Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010161932Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010195762Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010247354Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010298555Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010332056Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010417168Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01049157Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010555211Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010603132Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010657064Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010695855Z namespaces/openshift-network-node-identity/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010782117Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010827648Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010883139Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies/default-deny.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010935051Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010943441Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.010984632Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.011127315Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01130824Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.011448193Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.011649468Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.011829062Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.011856353Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.011947855Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.011958775Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.011969196Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012026207Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012069108Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012119539Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01214677Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012202631Z namespaces/openshift-network-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012249142Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012310824Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012338934Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012392056Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012426997Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012536059Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012614141Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012692423Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012757755Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012798996Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012880928Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.012913979Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01296912Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013004751Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013085223Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013127914Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013171985Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013251987Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013301448Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013310398Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013324309Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01339206Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013431601Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013551254Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013630896Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01378765Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.013856461Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014054236Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014137248Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014266801Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014320823Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014370254Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014442516Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014507807Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014565209Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01461295Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014654701Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014714242Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014748373Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014796164Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014840245Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.014939608Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01501644Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015060741Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015103632Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015160413Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015195744Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015252355Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015286866Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015341567Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015384039Z namespaces/openshift-network-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01546601Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015512562Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015560033Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies/iptables-alerter.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015609174Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015617594Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015656915Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015797219Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.015994943Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016206948Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016408244Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016601168Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016609168Z namespaces/openshift-network-operator/pods/iptables-alerter-5f2j9/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016649949Z namespaces/openshift-network-operator/pods/iptables-alerter-5f2j9/iptables-alerter-5f2j9.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016723871Z namespaces/openshift-network-operator/pods/iptables-alerter-5f2j9/iptables-alerter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016734831Z namespaces/openshift-network-operator/pods/iptables-alerter-5f2j9/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016741372Z namespaces/openshift-network-operator/pods/iptables-alerter-5f2j9/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016770002Z namespaces/openshift-network-operator/pods/iptables-alerter-5f2j9/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.016875635Z namespaces/openshift-network-operator/pods/iptables-alerter-5f2j9/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017014078Z namespaces/openshift-network-operator/pods/iptables-alerter-5f2j9/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017053259Z namespaces/openshift-network-operator/pods/iptables-alerter-h6x2s/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01709269Z namespaces/openshift-network-operator/pods/iptables-alerter-h6x2s/iptables-alerter-h6x2s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017159272Z namespaces/openshift-network-operator/pods/iptables-alerter-h6x2s/iptables-alerter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017170322Z namespaces/openshift-network-operator/pods/iptables-alerter-h6x2s/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017176472Z namespaces/openshift-network-operator/pods/iptables-alerter-h6x2s/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017205233Z namespaces/openshift-network-operator/pods/iptables-alerter-h6x2s/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017291415Z namespaces/openshift-network-operator/pods/iptables-alerter-h6x2s/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017368977Z namespaces/openshift-network-operator/pods/iptables-alerter-h6x2s/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017392607Z namespaces/openshift-network-operator/pods/iptables-alerter-vhkmj/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017435908Z namespaces/openshift-network-operator/pods/iptables-alerter-vhkmj/iptables-alerter-vhkmj.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01750514Z namespaces/openshift-network-operator/pods/iptables-alerter-vhkmj/iptables-alerter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01751337Z namespaces/openshift-network-operator/pods/iptables-alerter-vhkmj/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01751827Z namespaces/openshift-network-operator/pods/iptables-alerter-vhkmj/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017540341Z namespaces/openshift-network-operator/pods/iptables-alerter-vhkmj/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017642083Z namespaces/openshift-network-operator/pods/iptables-alerter-vhkmj/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017709565Z namespaces/openshift-network-operator/pods/iptables-alerter-vhkmj/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017726556Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017778737Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017831648Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017851288Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01791533Z namespaces/openshift-node/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017932881Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017947161Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.017988182Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.018132855Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.018306159Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.018443523Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.018657778Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.018846803Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.018905714Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.018970736Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019011897Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019057728Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019097659Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019183911Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019260063Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019339375Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019376176Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019417306Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019469588Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019517199Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019589781Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019637982Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019674543Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019755595Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019804276Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.019843327Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.01996314Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020044142Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020123634Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020200445Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020280487Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.02039966Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020587475Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020653407Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020691248Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020746999Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.02078422Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020827691Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020871082Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.020977074Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021048916Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021099237Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021135078Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021184909Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.02122077Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021318993Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021354244Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021438866Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021516017Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021524388Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021561139Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021712102Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.021983689Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022119042Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022318067Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022501241Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022555013Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022622724Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022657125Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022700676Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022738767Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022792688Z namespaces/openshift-operators/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022799739Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.022803719Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0228399Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023018564Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023215349Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023331982Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023544697Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023717301Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023761572Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023827874Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023847514Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023941766Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.023962547Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024016738Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024057169Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024238124Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024317605Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024396938Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024454379Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0244893Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024620363Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024654894Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024709735Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024752056Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024834538Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024894019Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.024944121Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025024513Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025079934Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025087154Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025117785Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025174206Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025217207Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025405342Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025494324Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.025878943Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.026095819Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.026584441Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.026668333Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.026886378Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.026929259Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.02697011Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027056212Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027172995Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027190455Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027299958Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027338199Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.02737238Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027430801Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027476122Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027524523Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027569355Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027607645Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027615396Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027658297Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027712908Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027752519Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027829281Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027934453Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.027978074Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028012135Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028069137Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028104477Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028183659Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028225131Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028280952Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028288242Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028316013Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028400905Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028448356Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028488917Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028567719Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.02861561Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028645641Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies/default-deny.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028701942Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028713102Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028751063Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.028925997Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.029136573Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.029329057Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.029534772Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.029727187Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.029736347Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.029797049Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovnkube-node-7ppff.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.029981873Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-node/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.029994614Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030001224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030029004Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030297251Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030352862Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030439214Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030448334Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030452325Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030497476Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030606709Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03068194Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030722161Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/nbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030732332Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/nbdb/nbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030746282Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/nbdb/nbdb/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030755912Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/nbdb/nbdb/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030886865Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030962557Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.030993198Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/northd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031000248Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/northd/northd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031006278Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/northd/northd/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031043529Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/northd/northd/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031151462Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031216923Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/northd/northd/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031263014Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-acl-logging/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031275045Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031281585Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031293215Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031383247Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031455679Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0314827Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03149009Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-controller/ovn-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03149581Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.031552961Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.032011092Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.032124745Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.032150826Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovnkube-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.032160866Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.032166996Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.032196017Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03437236Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034436582Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034473792Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/sbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034481693Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/sbdb/sbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034485713Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/sbdb/sbdb/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034534284Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/sbdb/sbdb/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034633746Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034701658Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7ppff/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034745279Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034803571Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovnkube-node-82mjm.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034974564Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-node/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034985375Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.034989645Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035017036Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035130718Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03519897Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035240911Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035250691Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035255241Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035283092Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035388614Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035458736Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035496677Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/nbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035504807Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/nbdb/nbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035508877Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/nbdb/nbdb/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035539778Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/nbdb/nbdb/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035641701Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035710522Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035741513Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/northd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035747223Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/northd/northd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035751003Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/northd/northd/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035788224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/northd/northd/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.035929028Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03601189Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/northd/northd/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03602902Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-acl-logging/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03603591Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03604425Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036085551Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036188604Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036251155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036281356Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036286266Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-controller/ovn-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036294907Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036339698Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036698826Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036757938Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036797639Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovnkube-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036804279Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.036808259Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03684089Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.038843559Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.038936611Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.038972112Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/sbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.038981012Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/sbdb/sbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.038985852Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/sbdb/sbdb/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039017583Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/sbdb/sbdb/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039117335Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039183477Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82mjm/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039223818Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039283929Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovnkube-node-hdm75.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039452843Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-node/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039460344Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039464264Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039497745Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039608857Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039681709Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03972079Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03972738Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.03973129Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039760451Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039890544Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.039966736Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040009567Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/nbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040017637Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/nbdb/nbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040021507Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/nbdb/nbdb/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040057578Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/nbdb/nbdb/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04016397Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040225802Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040265933Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/northd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040272753Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/northd/northd/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040276523Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/northd/northd/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040311644Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/northd/northd/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040423587Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040498999Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/northd/northd/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04052833Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-acl-logging/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04053818Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04054969Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04057006Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040662913Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040736804Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040778885Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040787656Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-controller/ovn-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040791686Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.040818437Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.041194176Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.041252047Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.041287538Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovnkube-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.041297648Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.041301948Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.041335249Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.042926768Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.042994009Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04303736Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/sbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043053241Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/sbdb/sbdb/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043058521Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/sbdb/sbdb/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043079501Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/sbdb/sbdb/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043180834Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043248316Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-hdm75/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043292517Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043313957Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043365918Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043376949Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0434099Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043507672Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043641425Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043695096Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043733707Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04385856Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.043973643Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044038105Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044085666Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044129777Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044137187Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044141247Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044170508Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044223659Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044231629Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04426958Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044473085Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04468944Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.044833894Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04505862Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045277505Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045324636Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045374827Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045416938Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04547552Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045512871Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045591193Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045710905Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045819298Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045852069Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045925781Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.045960062Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046007573Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046087374Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046136256Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046174367Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046254289Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04629716Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04630634Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046345551Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046395282Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046432553Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046579026Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046661408Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046878064Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.046958176Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047074049Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047193911Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047376276Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047444327Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047487719Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04753712Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047578061Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047622842Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047666313Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047743075Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047821097Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047889168Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047925269Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.047977151Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048013921Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048073433Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048119924Z namespaces/openshift-service-ca-operator/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048197156Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048242667Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048254577Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048292088Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048437282Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048623376Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04875695Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.048987055Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049161809Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049169709Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-544d749c8c-2jl5h/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.04920661Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-544d749c8c-2jl5h/service-ca-operator-544d749c8c-2jl5h.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049282112Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-544d749c8c-2jl5h/service-ca-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049289252Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-544d749c8c-2jl5h/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049293212Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-544d749c8c-2jl5h/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049328513Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-544d749c8c-2jl5h/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049475707Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-544d749c8c-2jl5h/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049621111Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049659241Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049705002Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049745794Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049790424Z namespaces/openshift-service-ca/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049834686Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049907127Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049945838Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.049994209Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05003475Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050117532Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050231875Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050322177Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050361148Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050402409Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0504473Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050494262Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050584474Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050626305Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050670896Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050747478Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050780288Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050789449Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05083337Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050954183Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.050991114Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051102276Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051182048Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051293011Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051370753Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051482585Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051604879Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051773653Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051832334Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051881085Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051947727Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.051987688Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052032269Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05207649Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052155042Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052234124Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052282465Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052324256Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052369667Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052410348Z namespaces/openshift-service-ca/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05248999Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052532401Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052540791Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052580322Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052723616Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.052916491Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053063014Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053249809Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053527555Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053535266Z namespaces/openshift-service-ca/pods/service-ca-5fffc97bdf-qlkzv/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053571676Z namespaces/openshift-service-ca/pods/service-ca-5fffc97bdf-qlkzv/service-ca-5fffc97bdf-qlkzv.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053678819Z namespaces/openshift-service-ca/pods/service-ca-5fffc97bdf-qlkzv/service-ca-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053689689Z namespaces/openshift-service-ca/pods/service-ca-5fffc97bdf-qlkzv/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053694939Z namespaces/openshift-service-ca/pods/service-ca-5fffc97bdf-qlkzv/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05373222Z namespaces/openshift-service-ca/pods/service-ca-5fffc97bdf-qlkzv/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.053979396Z namespaces/openshift-service-ca/pods/service-ca-5fffc97bdf-qlkzv/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054044598Z namespaces/openshift-service-ca/pods/service-ca-5fffc97bdf-qlkzv/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054086489Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0541215Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054168161Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054210212Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054254533Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054299514Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054359635Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054399916Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054444768Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054483158Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054580021Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054659603Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054743435Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054789116Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054831017Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054878078Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.054938089Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055013091Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055057072Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055099864Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055174635Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055219826Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055262757Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05536608Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055443152Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055522954Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055601996Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055684688Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055814171Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.055996545Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056045556Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056085248Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056132189Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0561739Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056219921Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056260062Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056340624Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056419796Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056465197Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056506208Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056552349Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05659498Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056677012Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056724513Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056732573Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056772804Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.056978689Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.057153434Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.057293177Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.057491472Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.057687656Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.057730177Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.057856611Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.057912992Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.057956803Z namespaces/openshift/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058011524Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058052575Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058100177Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058145507Z namespaces/openshift/apps/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058180539Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05826535Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058348012Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058424214Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058469045Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058504696Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058570168Z namespaces/openshift/batch/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058609099Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058686761Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058731712Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058773993Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058855905Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058919056Z namespaces/openshift/core/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.058952557Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.05905896Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059134091Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059208883Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059291845Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059367787Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0594856Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059645734Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059694525Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059736966Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059783627Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.059826518Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.061228073Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.061253623Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.061355625Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.061439608Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-aspnet.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.061563431Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.061706374Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.061839267Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.061988841Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.062125864Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.062251877Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.06238597Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.062523494Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.062624466Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.06275169Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.062913133Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063030666Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063120378Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063216171Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063318763Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063525798Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063641821Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063740734Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063836296Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.063965739Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064065831Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064157824Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064256706Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064349158Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064449841Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064558203Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064666046Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064767999Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.064883971Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065071766Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065161548Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065270961Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065357083Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065475756Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065622709Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065715261Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065836144Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.065963928Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.06608485Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.066207993Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.066299125Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.066392768Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.066528981Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.066659254Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.066798298Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.066935581Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.067063354Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.067160406Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.067250309Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.067338251Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.067461934Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.067597937Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.06771214Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.067829283Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.067947806Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068038638Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068165751Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068241643Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068294004Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068373896Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068451388Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068497859Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.06854428Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068588911Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068630292Z namespaces/openshift/networking.k8s.io/ingresses.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068704474Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068748545Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068757296Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.068790296Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.069011891Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.1.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.069201896Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.069428432Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.069624937Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.069808461Z namespaces/openshift/policy/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.069849242Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.069919593Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.069957934Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070005816Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070012556Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070048987Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070159699Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070282432Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070389875Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070496947Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070669802Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070765274Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.070901577Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.07101295Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.071137853Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.071245966Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.071357068Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.071481662Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.071619215Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.071722208Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.071849771Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.071998304Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.072108807Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.07225929Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.072366243Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.072471136Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.072570388Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.072701321Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.072841405Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073032589Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073128702Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073235174Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073327207Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073449989Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073563752Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073659304Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073768227Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.073858559Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074001073Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074116535Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074223748Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074333801Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074431183Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074566526Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074676029Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074768831Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.074917085Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.075037258Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.075160671Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.075283944Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.075408857Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.075552021Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.075682504Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.075809157Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.07595245Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076031302Z network_logs/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076072743Z network_logs/cluster_scale [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076164035Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076231027Z network_logs/multi-networkpolicy [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076311799Z network_logs/net-attach-def [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076391461Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076549405Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076643107Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076690458Z nodes/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076724819Z nodes/debug [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076811331Z nodes/ip-10-0-128-97.us-east-2.compute.internal/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.076843132Z nodes/ip-10-0-128-97.us-east-2.compute.internal/cpu_affinities.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.07719287Z nodes/ip-10-0-128-97.us-east-2.compute.internal/dmesg [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.077498398Z nodes/ip-10-0-128-97.us-east-2.compute.internal/ethtool_channels [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.07758237Z nodes/ip-10-0-128-97.us-east-2.compute.internal/ethtool_features [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.077667342Z nodes/ip-10-0-128-97.us-east-2.compute.internal/ip-10-0-128-97.us-east-2.compute.internal_logs_kubelet.gz [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.078376439Z nodes/ip-10-0-128-97.us-east-2.compute.internal/irq_affinities.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.078498822Z nodes/ip-10-0-128-97.us-east-2.compute.internal/lscpu [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.078727918Z nodes/ip-10-0-128-97.us-east-2.compute.internal/lspci [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.07881588Z nodes/ip-10-0-128-97.us-east-2.compute.internal/podresources.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.078942633Z nodes/ip-10-0-128-97.us-east-2.compute.internal/pods_info.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.079064096Z nodes/ip-10-0-128-97.us-east-2.compute.internal/proc_cmdline [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.079143808Z nodes/ip-10-0-128-97.us-east-2.compute.internal/sysinfo.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.07924205Z nodes/ip-10-0-128-97.us-east-2.compute.internal/sysinfo.tgz [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.079394724Z nodes/ip-10-0-132-192.us-east-2.compute.internal/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.079425295Z nodes/ip-10-0-132-192.us-east-2.compute.internal/cpu_affinities.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.079713011Z nodes/ip-10-0-132-192.us-east-2.compute.internal/dmesg [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.079990168Z nodes/ip-10-0-132-192.us-east-2.compute.internal/ethtool_channels [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.08007458Z nodes/ip-10-0-132-192.us-east-2.compute.internal/ethtool_features [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.080151752Z nodes/ip-10-0-132-192.us-east-2.compute.internal/ip-10-0-132-192.us-east-2.compute.internal_logs_kubelet.gz [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.080745797Z nodes/ip-10-0-132-192.us-east-2.compute.internal/irq_affinities.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.08088656Z nodes/ip-10-0-132-192.us-east-2.compute.internal/lscpu [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.081067315Z nodes/ip-10-0-132-192.us-east-2.compute.internal/lspci [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.081155957Z nodes/ip-10-0-132-192.us-east-2.compute.internal/podresources.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.081272249Z nodes/ip-10-0-132-192.us-east-2.compute.internal/pods_info.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.081385942Z nodes/ip-10-0-132-192.us-east-2.compute.internal/proc_cmdline [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.081470024Z nodes/ip-10-0-132-192.us-east-2.compute.internal/sysinfo.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.081552506Z nodes/ip-10-0-132-192.us-east-2.compute.internal/sysinfo.tgz [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.0816986Z nodes/ip-10-0-138-251.us-east-2.compute.internal/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.081740141Z nodes/ip-10-0-138-251.us-east-2.compute.internal/cpu_affinities.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.082096329Z nodes/ip-10-0-138-251.us-east-2.compute.internal/dmesg [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.082461018Z nodes/ip-10-0-138-251.us-east-2.compute.internal/ethtool_channels [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.08255008Z nodes/ip-10-0-138-251.us-east-2.compute.internal/ethtool_features [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.082631373Z nodes/ip-10-0-138-251.us-east-2.compute.internal/ip-10-0-138-251.us-east-2.compute.internal_logs_kubelet.gz [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.083416261Z nodes/ip-10-0-138-251.us-east-2.compute.internal/irq_affinities.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.083545625Z nodes/ip-10-0-138-251.us-east-2.compute.internal/lscpu [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.083626627Z nodes/ip-10-0-138-251.us-east-2.compute.internal/lspci [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.083711569Z nodes/ip-10-0-138-251.us-east-2.compute.internal/podresources.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.083817031Z nodes/ip-10-0-138-251.us-east-2.compute.internal/pods_info.json [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.083966115Z nodes/ip-10-0-138-251.us-east-2.compute.internal/proc_cmdline [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.084060247Z nodes/ip-10-0-138-251.us-east-2.compute.internal/sysinfo.log [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.084127279Z nodes/ip-10-0-138-251.us-east-2.compute.internal/sysinfo.tgz [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.084297013Z pod_network_connectivity_check/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.084335424Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.084438377Z static-pods/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.084446017Z static-pods/kube-apiserver/ [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.110483569Z [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.11050568Z sent 74,583 bytes received 7,867,613 bytes 5,294,797.33 bytes/sec [must-gather] [must-gather-6hmfk] OUT 2026-10-30T23:45:42.1105118Z total size is 120,709,511 speedup is 15.20 [must-gather] [must-gather ] OUT 2026-10-30T23:45:42.560978725Z namespace/openshift-must-gather-dn4ld deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 14dd1418-000a-44c2-b6a2-41b3111578dc [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.22.20" [must-gather] ClusterOperators: [must-gather] clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-5qxqp [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-5qxqp-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-5qxqp [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 15853899 Oct 30 23:47 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-5qxqp/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 90a1137] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-5qxqp/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 567e0e0..90a1137 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-10-30T23:48:02.287Z","type":3}]