task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-p45rk-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-p45rk-kubeconfig [get-kubeconfig] Found admin password secret: cluster-p45rk-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-p45rk-password [get-kubeconfig] API Server URL: https://aa9909fac3272479a8f8d8923d568884-e2d1fad4d448ae5c.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.043e8889-7773-493c-bcb4-d7808116c490.prod.konfluxeaas.com [clone-repo] work/RHOAIENG-79440 [clone-repo] https://github.com/jlost/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 85% (2793/3251) Updating files: 86% (2796/3251) Updating files: 87% (2829/3251) Updating files: 88% (2861/3251) Updating files: 89% (2894/3251) Updating files: 90% (2926/3251) Updating files: 91% (2959/3251) Updating files: 92% (2991/3251) Updating files: 93% (3024/3251) Updating files: 94% (3056/3251) Updating files: 95% (3089/3251) Updating files: 96% (3121/3251) Updating files: 97% (3154/3251) Updating files: 98% (3186/3251) Updating files: 99% (3219/3251) Updating files: 100% (3251/3251) Updating files: 100% (3251/3251), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1821 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1821 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1821 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Prepared 101 packages in 1.81s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 423ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 47ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://aa9909fac3272479a8f8d8923d568884-e2d1fad4d448ae5c.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (50/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-gjlbf 1/1 Running 0 42s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-gjlbf condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-rt8c7 1/1 Running 0 46s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-rt8c7 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-b5dz8 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-b5dz8 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:d2cb06ca9eeb023462e079ca78f69f867aac7089d74cc32b24351cfb0f653480 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:2d78b1e75ce4a3e633713c1f964bde1c2425164638b26ad77af20d6340c844d7 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-545d98d4c7-4jhnx 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-545d98d4c7-4jhnx condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-4tnc8 1/1 Running 0 12s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-4tnc8 condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-x4v6g 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-x4v6g condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-8699b59bbc-nvf9s 0/1 ContainerCreating 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-nq65w 1/1 Running 0 63s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-nq65w condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-nq65w 1/1 Running 0 73s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-nq65w condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-p45rk-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-p45rk-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-p45rk-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-p45rk-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-p45rk-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-p45rk-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Tue Jul 28 17:20:27 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-ftlgf condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-cd4d44c67-vp66g" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-cd4d44c67-d9v9x condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:147fe95b95d608b1ded94d3020e1b5a837760c7d296c09f9beafaa934c1fe9d7 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:b5c835d640b650f164ba7f36d7008760c88e2974efa65853ee167323c441580c [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:db3f49e79083b8d5ad19a51f3825c824dc9a5a14f33cff392210d0a6aa1c71ad [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:77c0c8e0b30f40efcf2f0417b910a58f8b07503b5e75a61bdb9ab9efab31d46f [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.124.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'cov': '5.0.0', 'xdist': '3.6.1', 'anyio': '4.9.0', 'httpx': '0.30.0', 'metadata': '3.1.1', 'json-report': '1.5.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-0.23.8, cov-5.0.0, xdist-3.6.1, anyio-4.9.0, httpx-0.30.0, metadata-3.1.1, json-report-1.5.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] 2026-07-28 17:22:58.859 6528 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-28 17:22:58.860 6525 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-28 17:25:43.036 6525 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 17:25:43.036 6525 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 17:25:43.064 6525 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:25:43.064 6525 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:25:43.070 6525 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:25:43.070 6525 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-28 17:25:54.016 6525 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 17:25:54.016 6525 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 17:25:54.039 6525 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:25:54.039 6525 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:25:54.050 6525 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 17:25:54.050 6525 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_leave_group [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_leave_group [e2e-llm-inference-service] llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_three_member_group [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_three_member_group [e2e-llm-inference-service] llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_late_join [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_late_join [e2e-llm-inference-service] llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_delete_at_nonzero_weight [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_delete_at_nonzero_weight [e2e-llm-inference-service] llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_rollback [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_rollback [e2e-llm-inference-service] llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_force_stop_route_owner [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_canary_lifecycle.py::TestCanaryLifecycle::test_force_stop_route_owner [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-28 18:31:34.984 6528 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-6f0aa991 [e2e-llm-inference-service] 2026-07-28 18:31:34.984 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-6f0aa991 [e2e-llm-inference-service] 2026-07-28 18:31:35.008 6528 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 18:31:35.008 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 18:31:35.016 6528 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 18:31:35.016 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-28 18:38:47.881 6528 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-07-28 18:38:47.881 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-07-28 18:38:47.908 6528 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 18:38:47.908 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 18:38:47.916 6528 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 18:38:47.916 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 18:38:47.916 6528 kserve.trace Checking HttpRoute router-route-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-07-28 18:38:47.916 6528 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-07-28 18:38:47.919 6528 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-28 18:38:47.919 6528 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-28 18:38:47.935 6528 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-28 18:38:47.935 6528 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-28 18:38:47.935 6528 kserve.trace Checking HttpRoute router-route-2 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-07-28 18:38:47.935 6528 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-07-28 18:38:47.942 6528 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-28 18:38:47.942 6528 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-28 18:38:47.951 6528 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-28 18:38:47.951 6528 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-28 18:53:03.529 6528 kserve.trace Checking Gateway router-gateway-2 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-07-28 18:53:03.529 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-07-28 18:53:03.554 6528 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-28 18:53:03.554 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-28 18:53:03.560 6528 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-28 18:53:03.560 6528 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-28 18:53:03.561 6528 kserve.trace Checking HttpRoute router-route-3 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-07-28 18:53:03.561 6528 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-07-28 18:53:03.564 6528 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-28 18:53:03.564 6528 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-28 18:53:03.573 6528 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-28 18:53:03.573 6528 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-28 18:53:03.573 6528 kserve.trace Checking HttpRoute router-route-4 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-07-28 18:53:03.573 6528 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-07-28 18:53:03.579 6528 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-28 18:53:03.579 6528 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-28 18:53:03.588 6528 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-28 18:53:03.588 6528 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] ___________ test_llm_auth_invalid_token_rejected[auth-invalid-token] ___________ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name=... {'name': 'model-fb-opt-125m-auth-invalid-20a12e51'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.auth [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="auth-invalid-token-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="auth-invalid-token", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_auth_invalid_token_rejected(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] """ [e2e-llm-inference-service] Test that when auth is enabled: [e2e-llm-inference-service] - Requests with MALFORMED tokens are rejected [e2e-llm-inference-service] """ [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] sa_name = f"{service_name}-test-sa" [e2e-llm-inference-service] ns = test_case.llm_service.metadata.namespace [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Enable auth for this test [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "true" [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Create LLMInferenceService [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_auth.py:440: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...invali-efc9e473'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-invalid-20a12e51'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T17:30:30.884470', start_time = 1785259830.8848135 [e2e-llm-inference-service] duration = 900.2377202510834, timestamp_end = '2026-07-28T17:45:31.122542' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....-auth-invali-efc9e473'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-invalid-20a12e51'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fc258c50c20> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-auth-invalid-tok-7c58bb00 in namespace e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-auth-invalid-tok-7c58bb00 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-auth-invalid-tok-7c58bb00 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-single-cpu-auth-invali-efc9e473 in namespace e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-auth-invali-efc9e473 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-auth-invali-efc9e473 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-fb-opt-125m-auth-invalid-20a12e51 in namespace e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-auth-invalid-20a12e51 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-auth-invalid-20a12e51 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_auth_invalid_token_rejected] [2026-07-28T17:30:30.721756] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='auth-invalid-token-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-invalid-token-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-invalid-tok-7c58bb00'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-invali-efc9e473'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-invalid-20a12e51'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T17:30:30.734550] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'true'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-invalid-token-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-invalid-tok-7c58bb00'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-invali-efc9e473'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-invalid-20a12e51'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T17:30:30.884316] end - ✅ in 0.150s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T17:30:30.884470] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'true'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'auth-invalid-token-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-auth-invalid-tok-7c58bb00'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-auth-invali-efc9e473'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-auth-invalid-20a12e51'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:30:45Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'message': 'Inference Pool e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 2: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 3: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 4: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T17:45:31.122542] end - ❌ 900.238s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_auth.py:483 ❌ ERROR: Failed test for auth-invalid-token-test: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'auth-invalid-token-test' in 'e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService auth-invalid-token-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'true' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:30Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:30Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:30Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:31:17Z' [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] resourceVersion: '30097' [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-auth-invalid-tok-7c58bb00 [e2e-llm-inference-service] - name: workload-single-cpu-auth-invali-efc9e473 [e2e-llm-inference-service] - name: model-fb-opt-125m-auth-invalid-20a12e51 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a1277ed6ed4d54a79bb55bae173c59ec-1499523350.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a1277ed6ed4d54a79bb55bae173c59ec-1499523350.us-east-1.elb.amazonaws.com/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a1277ed6ed4d54a79bb55bae173c59ec-1499523350.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:30:45Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:17Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:17Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a1277ed6ed4d54a79bb55bae173c59ec-1499523350.us-east-1.elb.amazonaws.com/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 Normal Scheduled Successfully assigned e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq to ip-10-0-137-238.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:44 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:44 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 kubelet/ip-10-0-137-238.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:44 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 kubelet/ip-10-0-137-238.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:44 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 kubelet/ip-10-0-137-238.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-57cd4b8d86-7dpxq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxd9q [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 Normal Scheduled Successfully assigned e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxd9q to ip-10-0-133-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:44 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:44 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 kubelet/ip-10-0-133-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:44 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 kubelet/ip-10-0-133-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:44 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 kubelet/ip-10-0-133-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-57cd4b8d86 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-auth-invalid-token6000ef13b23b3f7ab1b02b4ac1c6ee1f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:30:43 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:31:04 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:31:03 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 17:31:03 e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod auth-invalid-token-test-kserve-57cd4b8d86-7dpxq (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 2026-07-28 17:30:44.505 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-07-28 17:30:44.506 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_6078d7df-c02b-4db4-804e-8749a2a52b57'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_2ed6881d-4217-4c1e-b562-5ba4cdb24150'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_7e2c7c4a-6245-48e5-b54f-6b33a793d9df'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_4d6be9c6-76b2-4ef0-bb8f-dc9b40c2951b'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/gPcsVCQDYDHk-_n0G9uADl7PXIM=.61c60ec52ed43038fff0fbbd68b080c94b0d94b4c8458dbd65965f9b17631c89.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_237f8f5f-e7d7-4a37-8723-2160ee22f5c4'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ce6897d1-74c5-493d-8052-37fcc09c4087'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_537c02f5-5160-4b46-a4ae-341dff32098c'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'aa0d86c2-c54d-4a2c-a61d-32130106d907', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 17:45:31 GMT', 'Content-Length': '229'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxd9q (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785259844.5249302,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5250518,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n- parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"http","pool-group":"inference.networking.k8s.io","pool-name":"auth-invalid-token-test-inference-pool","pool-namespace":"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5251942,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5257206,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: prefix-cache-scorer} {Type: no-hit-lru-scorer} {Type: max-score-picker} {Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5257528,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.526012,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"no-hit-lru-scorer","parameters":null,"type":"no-hit-lru-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"no-hit-lru-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5442214,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5442972,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5443292,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5443647,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5444226,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.544442,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5444868,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: no-hit-lru-scorer, Type: no-hit-lru-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.544564,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5446234,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5446417,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5446649,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5447497,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5447595,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00082f100 DataConfig:{Sources:[{Plugin:0xc00049dcb0 Extractors:[0xc00082f300]}]} FlowControlConfig: ParserRegistry:0xc00082f840}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5619001,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.562074,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5620918,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5620968,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5621064,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5621107,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5621157,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.56212,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5621276,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5621333,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5621376,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.562142,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5621526,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5621576,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5622118,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5622206,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5622346,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5622506,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5622606,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5624037,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5624187,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5625489,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5630252,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785259844.563674,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5637734,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785259844.563945,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5639722,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.564103,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5639908,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5649884,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5650754,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5680454,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5684044,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5684361,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.5709484,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.6654444,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.6654682,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.6654913,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785259844.6654627,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.6654954,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785259844.6655986,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785259844.665433,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785259844.6656213,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785259844.6657405,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"auth-invalid-token-test-inference-pool","namespace":"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6"},"namespace":"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6","name":"auth-invalid-token-test-inference-pool","reconcileID":"0a7cdf8d-f253-410f-89a1-85ca739a2072"} [e2e-llm-inference-service] {"level":"info","ts":1785259863.9441826,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"auth-invalid-token-test-inference-pool","namespace":"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6"},"namespace":"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6","name":"auth-invalid-token-test-inference-pool","reconcileID":"639bda85-b42e-424e-a346-4c969f59f525"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 677de1bf-5406-47b3-acd8-2e3090ce0080 [e2e-llm-inference-service] resourceVersion: '30035' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.35 [e2e-llm-inference-service] nodeName: ip-10-0-133-215.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxd9q [e2e-llm-inference-service] uid: 74660d04-9529-4f20-b4a8-f877de2307e7 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: b8c751d3-2f7a-414c-adfb-fe5207278357 [e2e-llm-inference-service] resourceVersion: '29344' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - notReadyAddresses: [e2e-llm-inference-service] - ip: 10.133.0.41 [e2e-llm-inference-service] nodeName: ip-10-0-137-238.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-57cd4b8d86-7dpxq [e2e-llm-inference-service] uid: d8797f78-2333-45d2-aabe-755fd05d755e [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-57cd4b8d86-7dpxq [e2e-llm-inference-service] generateName: auth-invalid-token-test-kserve-57cd4b8d86- [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: d8797f78-2333-45d2-aabe-755fd05d755e [e2e-llm-inference-service] resourceVersion: '29342' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 57cd4b8d86 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.41/23"],"mac_address":"0a:58:0a:85:00:29","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.41/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.41\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:29\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-57cd4b8d86 [e2e-llm-inference-service] uid: 7a7f2d52-e537-48b9-81a5-1fd5d234b7f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-137-238 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7a7f2d52-e537-48b9-81a5-1fd5d234b7f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.41"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-qrj7v [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-qrj7v [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000780000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-qrj7v [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000780000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-137-238.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c28,c12 [e2e-llm-inference-service] fsGroup: 1000780000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5svxx [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [storage-initializer]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] hostIP: 10.0.137.238 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.137.238 [e2e-llm-inference-service] podIP: 10.133.0.41 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.41 [e2e-llm-inference-service] startTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] containerID: cri-o://7fc8850720f50f1b56650fb3f28676624d3806a090801e459c07b28775585c3a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-qrj7v [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000780000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000780000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-qrj7v [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxd9q [e2e-llm-inference-service] generateName: auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc- [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 74660d04-9529-4f20-b4a8-f877de2307e7 [e2e-llm-inference-service] resourceVersion: '30033' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6d845bb7dc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.35/23"],"mac_address":"0a:58:0a:86:00:23","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.35/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.35\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:23\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc [e2e-llm-inference-service] uid: 36bbe656-e7d8-4413-9d52-f53d830f7f77 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-133-215 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"36bbe656-e7d8-4413-9d52-f53d830f7f77"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.35"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-zk6p9 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-invalid-token-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n -\ [e2e-llm-inference-service] \ pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-zk6p9 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000780000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-133-215.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c28,c12 [e2e-llm-inference-service] fsGroup: 1000780000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5svxx [e2e-llm-inference-service] - name: auth-invalid-token-test-epp-sa-dockercfg-cqmzx [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:45Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] hostIP: 10.0.133.215 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.133.215 [e2e-llm-inference-service] podIP: 10.134.0.35 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.35 [e2e-llm-inference-service] startTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://49d42dd9fcb70c46883333f99ba1d55c0c6cef1fa608c51ef016f8d8023ac15a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-zk6p9 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000780000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000780000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 4744ca40-06f2-4f3e-91a7-36775994bde3 [e2e-llm-inference-service] resourceVersion: '29230' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: auth-invalid-token-test-epp-sa-dockercfg-cqmzx [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"auth-invalid-token-test-epp-sa-dockercfg-cqmzx"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-5svxx"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-5svxx [e2e-llm-inference-service] - name: auth-invalid-token-test-epp-sa-dockercfg-cqmzx [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5svxx [e2e-llm-inference-service] - name: auth-invalid-token-test-epp-sa-dockercfg-cqmzx [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: f7a09fb0-4f77-48c7-9e4b-4ddfafd81370 [e2e-llm-inference-service] resourceVersion: '29252' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.186.213 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.186.213 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 1856e13a-55ce-47f5-85de-5252775b9cb0 [e2e-llm-inference-service] resourceVersion: '29222' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.116.151 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.116.151 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 2e10e37b-cbef-4612-8a22-8215ac5cd1ec [e2e-llm-inference-service] resourceVersion: '39535' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:40:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:40:44Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:40:44Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "auth-invalid-token-test-kserve-57cd4b8d86" has timed out [e2e-llm-inference-service] progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 4188dcfc-f493-455c-b176-a51ce3795bbd [e2e-llm-inference-service] resourceVersion: '30037' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-invalid-token-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-57cd4b8d86 [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 7a7f2d52-e537-48b9-81a5-1fd5d234b7f4 [e2e-llm-inference-service] resourceVersion: '29237' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 57cd4b8d86 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: auth-invalid-token-test-kserve [e2e-llm-inference-service] uid: 2e10e37b-cbef-4612-8a22-8215ac5cd1ec [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2e10e37b-cbef-4612-8a22-8215ac5cd1ec"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 57cd4b8d86 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 57cd4b8d86 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:3049312fa55515cbc94d5484b2c3ba6994735bde07ec862ad9b359b89e6c0af0 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-router-scheduler-6d845bb7dc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 36bbe656-e7d8-4413-9d52-f53d830f7f77 [e2e-llm-inference-service] resourceVersion: '30036' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6d845bb7dc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 4188dcfc-f493-455c-b176-a51ce3795bbd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4188dcfc-f493-455c-b176-a51ce3795bbd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6d845bb7dc [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6d845bb7dc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - auth-invalid-token-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] serviceAccount: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 5eaf4312-8028-47e5-85c2-e8bbb2d093d3 [e2e-llm-inference-service] resourceVersion: '29246' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: auth-invalid-token-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: fadb17ad-d8fe-4af2-aeec-0aacec04418b [e2e-llm-inference-service] resourceVersion: '29244' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-epp-service-b7wsg [e2e-llm-inference-service] generateName: auth-invalid-token-test-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: a6e38631-552d-4512-9543-fe69fcf94025 [e2e-llm-inference-service] resourceVersion: '30034' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: auth-invalid-token-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-epp-service [e2e-llm-inference-service] uid: f7a09fb0-4f77-48c7-9e4b-4ddfafd81370 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:31:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f7a09fb0-4f77-48c7-9e4b-4ddfafd81370"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.35 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxd9q [e2e-llm-inference-service] uid: 74660d04-9529-4f20-b4a8-f877de2307e7 [e2e-llm-inference-service] nodeName: ip-10-0-133-215.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc-jv6pl [e2e-llm-inference-service] generateName: auth-invalid-token-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: b1b2a88d-3e24-4ba6-9563-662edea27688 [e2e-llm-inference-service] resourceVersion: '29345' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] uid: 1856e13a-55ce-47f5-85de-5252775b9cb0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1856e13a-55ce-47f5-85de-5252775b9cb0"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.41 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] serving: false [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-57cd4b8d86-7dpxq [e2e-llm-inference-service] uid: d8797f78-2333-45d2-aabe-755fd05d755e [e2e-llm-inference-service] nodeName: ip-10-0-137-238.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: 5eaf4312-8028-47e5-85c2-e8bbb2d093d3 [e2e-llm-inference-service] resourceVersion: '29246' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6:auth-invalid-token-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] name: auth-invalid-token-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] name: auth-invalid-token-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] uid: fadb17ad-d8fe-4af2-aeec-0aacec04418b [e2e-llm-inference-service] resourceVersion: '29244' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:31:03Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29757' [e2e-llm-inference-service] uid: 5edc1249-d646-4391-8fa8-80293fb5a50b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:31:03Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29757' [e2e-llm-inference-service] uid: 5edc1249-d646-4391-8fa8-80293fb5a50b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T17:31:03Z' [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29747' [e2e-llm-inference-service] uid: b5379ed9-48a8-4122-a208-8f7f6f3d5c95 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:03Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T17:31:03Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29297' [e2e-llm-inference-service] uid: 7f413994-a826-4104-ba61-258658961450 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-epp-service.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-epp-service.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29760' [e2e-llm-inference-service] uid: 10620c8d-b3a7-4959-97be-9bd625e020d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-inference-pool-ip-bbc77b0f.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29327' [e2e-llm-inference-service] uid: c1407d9c-ae77-4b10-93e1-658343ffa3f0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29297' [e2e-llm-inference-service] uid: 7f413994-a826-4104-ba61-258658961450 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-epp-service.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-epp-service.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29760' [e2e-llm-inference-service] uid: 10620c8d-b3a7-4959-97be-9bd625e020d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-inference-pool-ip-bbc77b0f.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29327' [e2e-llm-inference-service] uid: c1407d9c-ae77-4b10-93e1-658343ffa3f0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29297' [e2e-llm-inference-service] uid: 7f413994-a826-4104-ba61-258658961450 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-epp-service.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-epp-service.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:31:04Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29760' [e2e-llm-inference-service] uid: 10620c8d-b3a7-4959-97be-9bd625e020d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-inference-pool-ip-bbc77b0f.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:44Z' [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29327' [e2e-llm-inference-service] uid: c1407d9c-ae77-4b10-93e1-658343ffa3f0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: auth-invalid-token-test-kserve-workload-svc.e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17743c01-2f33-4f3c-a79a-0d10fdc387c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T17:30:43Z' [e2e-llm-inference-service] name: auth-invalid-token-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: auth-invalid-token-test [e2e-llm-inference-service] uid: 17743c01-2f33-4f3c-a79a-0d10fdc387c3 [e2e-llm-inference-service] resourceVersion: '29262' [e2e-llm-inference-service] uid: fc8509ec-f9bb-41b6-a4bb-c158df5d89a3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: auth-invalid-token-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: auth-invalid-token-test-kserve-router-scheduler-6d845bb7dclxd9q [e2e-llm-inference-service] namespace: e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T17:45:32Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: auth-invalid-token-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6d845bb7dc [e2e-llm-inference-service] timestamp: '2026-07-28T17:45:15Z' [e2e-llm-inference-service] window: 16.42s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 670645n [e2e-llm-inference-service] memory: 21052Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_auth.py:517 ⏭️ Skipping deletion of auth-invalid-token-test due to test failure (SKIP_DELETION_ON_FAILURE=True) [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_auth_invalid_token_rejected] [2026-07-28T17:45:32.501679] end - ❌ 901.780s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:30:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:31:17Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:31:04Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T17:48:05.284533', start_time = 1785260885.2849283 [e2e-llm-inference-service] duration = 900.7014257907867, timestamp_end = '2026-07-28T18:03:05.986374' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fc253f751c0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-28T17:48:04.824244] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T17:48:04.837135] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T17:48:05.284320] end - ✅ in 0.447s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T17:48:05.284533] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 5: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:48:18Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'message': 'HPA not yet visible in cache', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 6: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:03:05.986374] end - ❌ 900.701s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:03:05.986616] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-hd4zt'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-hd4zt'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-hd4zt'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-hd4zt'] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 8: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:03:26.127585] end - ✅ in 20.140s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-28T18:03:26.127758] end - ❌ 921.303s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T17:48:50Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T17:48:45Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_deployment(test_case: TestCase): [e2e-llm-inference-service] """KEDA + Deployment: ScaledObject exists with WVA annotations; no HPA; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:606: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:03:27.005951', start_time = 1785261807.0062244 [e2e-llm-inference-service] duration = 900.011447429657, timestamp_end = '2026-07-28T18:18:27.017674' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fc253f75a80> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_deployment] [2026-07-28T18:03:26.840263] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:03:26.853252] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:03:27.005813] end - ✅ in 0.152s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:03:27.005951] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:03:36Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'message': 'ScaledObject conditions not yet available', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 10: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:18:27.017674] end - ❌ 900.011s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:18:27.017798] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-bjf25'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-bjf25'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-bjf25'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-bjf25'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:18:47.117281] end - ✅ in 20.099s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_deployment] [2026-07-28T18:18:47.117404] end - ❌ 920.277s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:36Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:04:20Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:03:55Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_lws(test_case: TestCase): [e2e-llm-inference-service] """HPA + LWS: HPA exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:670: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...ale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:18:48.789897', start_time = 1785262728.7902377 [e2e-llm-inference-service] duration = 900.5734322071075, timestamp_end = '2026-07-28T18:33:49.363672' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fc253f76700> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_lws] [2026-07-28T18:18:48.316186] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:18:48.328797] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:18:48.789594] end - ✅ in 0.461s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:18:48.789897] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 12: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 13: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 14: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:33:49.363672] end - ❌ 900.573s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:33:49.363797] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:34:14.638177] end - ✅ in 25.274s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_lws] [2026-07-28T18:34:14.638383] end - ❌ 926.322s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:19:07Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:19:37Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:20Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:19:21Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_lws(test_case: TestCase): [e2e-llm-inference-service] """KEDA + LWS: ScaledObject exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:728: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T18:34:15.068504', start_time = 1785263655.0688283 [e2e-llm-inference-service] duration = 900.7306854724884, timestamp_end = '2026-07-28T18:49:15.799516' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fc253f774c0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_lws] [2026-07-28T18:34:14.988105] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T18:34:15.002218] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T18:34:15.068370] end - ✅ in 0.066s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T18:34:15.068504] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'ScaledObject conditions not yet available', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'RouterReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 15: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 16: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 17: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T17:23:26Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T18:49:15.799516] end - ❌ 900.731s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T18:49:15.799606] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T18:49:35.896580] end - ✅ in 20.097s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_lws] [2026-07-28T18:49:35.896704] end - ❌ 920.908s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'RouterReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T18:34:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T18:34:31Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T18:34:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T18:34:35Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-lws-e541a132 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:92: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 5 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 5 failed, 56 passed, 3 skipped, 32 warnings in 5589.63s (1:33:09) ======= [must-gather] [must-gather ] OUT 2026-07-28T18:56:08.022190197Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: ae3af18b-c19b-4ff5-9c6e-bc65d4c0a656 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T18:56:08.065469358Z namespace/openshift-must-gather-867wq created [must-gather] [must-gather ] OUT 2026-07-28T18:56:08.06978079Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-jmtc5 created [must-gather] [must-gather ] OUT 2026-07-28T18:56:08.092326854Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:14.951972315Z [disk usage checker] Started [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:14.955311595Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.225919799Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.395015797Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.395048572Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.395048572Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.398271308Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.571897893Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.571929271Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.571929271Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.574731720Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.743177644Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.743208407Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.743208407Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.745809929Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.908857420Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.908888166Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.908888166Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:15.911666637Z Error getting logs from rhods-notebooks [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.075709416Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.075825758Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.075825758Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.078627535Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.239647394Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.239677936Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.239677936Z namespaces "istio-system" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.242656359Z Error getting logs from istio-system [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.405352865Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.405383318Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.405383318Z namespaces "knative-serving" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.408166143Z Error getting logs from knative-serving [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.572374290Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.572407943Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.572407943Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.575533148Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:16.867410775Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.167329564Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.310514648Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.396524350Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.400356139Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.537982172Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.621218575Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.623838833Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.758208077Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.840042196Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.842645095Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:17.978728186Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.061681930Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.064324583Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.200246997Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.284120603Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.286797288Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.423521418Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.512624688Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.515605041Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.650260989Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.735680394Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.738694371Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.872005340Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.957387040Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:18.960314060Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.103786449Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.196172721Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.198897234Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.334629553Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.421588247Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.424471112Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.556889059Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.643619245Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.646834036Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.779798058Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.862468424Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.865100828Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.959901673Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:19.999935373Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.084252847Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.087024172Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.220562416Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.302999415Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.305676076Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.444111160Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.530691242Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.533594220Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.669766092Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.753259200Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.755958788Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.889211793Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.977330984Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:20.980450655Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:21.117128881Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:21.202484930Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:21.205347900Z Error collecting info from [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:21.932316522Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:22.060783571Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:22.190512104Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:22.332576198Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:22.463067481Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:23.070697910Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:23.203238603Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:23.340330116Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:23.482359700Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:23.615579708Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.188531818Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.327752288Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.408670288Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.480480259Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.646784073Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.783151032Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.868440132Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.946233993Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:24.964248453Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:25.115207756Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:25.247795078Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:25.324724590Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:25.394020624Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:25.578169707Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:25.722071764Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:25.800703347Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:25.874514062Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.042236328Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.181480413Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.256928945Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.331048580Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.502348986Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.635790503Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.715315140Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.784461440Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:26.948413546Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:27.084893716Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:27.159414839Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:27.234414663Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:27.398738810Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:27.534127611Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:27.608887015Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:27.685045182Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:27.871301585Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.016898904Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.099555296Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.173421537Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.341721270Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.483204416Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.561890158Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.633607629Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.795014851Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:28.929001482Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.003395671Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.074293407Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.246770895Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.383447253Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.458088498Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.533188099Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.703177671Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.832792120Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.906775436Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.970555748Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:29.977050786Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:30.143872109Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:30.281702644Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:30.360592536Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:30.430912919Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:30.600267225Z Gathering data for ns/e2e-test-llm-inference-service-b5dd93e6... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:30.747827838Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:30.856586787Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:30.937763110Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:31.100930928Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:31.668907732Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:31.814396833Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:31.934492213Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:32.104407806Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:32.796831642Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:33.035230241Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:33.205030791Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:33.368461554Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:34.016345958Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:34.185901337Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:34.339032605Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:34.501320737Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:34.975465928Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:35.183101978Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:35.358112243Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:35.479247851Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:35.643326797Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:36.330643707Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:36.511505284Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:36.641363040Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:36.802455900Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:37.535948230Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:37.715641591Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:37.846858458Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:38.008645177Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:38.749592507Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:38.921720799Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:39.052110432Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:39.216241935Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:39.900207430Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:39.979966308Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:40.071506404Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:40.199428986Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:40.362135087Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:41.061218650Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:41.235215578Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:41.362883655Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:41.529132297Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:42.214334260Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:42.386258659Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:42.514099960Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:42.682868853Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:43.385377719Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:43.561818152Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:43.683826583Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:43.845571829Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:44.544453972Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:44.721231509Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:44.851236534Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:44.984471807Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:45.015559907Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:45.720105010Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:45.899978658Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:46.029788667Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:46.207887825Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:46.908597684Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:47.091653137Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:47.221636669Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:47.399992069Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:48.094127035Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:48.267240122Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:48.410002046Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:48.580756102Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:49.289962383Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:49.468996287Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:49.599204914Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:49.765351240Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:49.989653504Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:50.522773306Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:50.715822477Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:50.856702849Z Wrote inspect data to must-gather. [must-gather] [must-gather-pjdrd] POD 2026-07-28T18:56:50.877706920Z Caches written to disk [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:51.888283047Z waiting for gather to complete [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:51.890850243Z downloading gather output [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.162612537Z receiving incremental file list [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.176549197Z ./ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.176613689Z aggregated-discovery-api.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.179575786Z aggregated-discovery-apis.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.180487679Z event-filter.html [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.181018483Z timestamp [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.181158607Z version [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182164373Z cluster-scoped-resources/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182177093Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182182063Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182217454Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182299866Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182314217Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182356308Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18244148Z namespaces/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18245171Z namespaces/e2e-test-llm-inference-service-b5dd93e6/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182481251Z namespaces/e2e-test-llm-inference-service-b5dd93e6/e2e-test-llm-inference-service-b5dd93e6.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182580324Z namespaces/e2e-test-llm-inference-service-b5dd93e6/apps.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182631785Z namespaces/e2e-test-llm-inference-service-b5dd93e6/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182698727Z namespaces/e2e-test-llm-inference-service-b5dd93e6/apps/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182775749Z namespaces/e2e-test-llm-inference-service-b5dd93e6/apps/daemonsets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.182876501Z namespaces/e2e-test-llm-inference-service-b5dd93e6/apps/deployments.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183053336Z namespaces/e2e-test-llm-inference-service-b5dd93e6/apps/replicasets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18322647Z namespaces/e2e-test-llm-inference-service-b5dd93e6/apps/statefulsets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183267171Z namespaces/e2e-test-llm-inference-service-b5dd93e6/autoscaling/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183309103Z namespaces/e2e-test-llm-inference-service-b5dd93e6/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183368244Z namespaces/e2e-test-llm-inference-service-b5dd93e6/batch/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183399295Z namespaces/e2e-test-llm-inference-service-b5dd93e6/batch/cronjobs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183520778Z namespaces/e2e-test-llm-inference-service-b5dd93e6/batch/jobs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183545978Z namespaces/e2e-test-llm-inference-service-b5dd93e6/build.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18361335Z namespaces/e2e-test-llm-inference-service-b5dd93e6/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183728503Z namespaces/e2e-test-llm-inference-service-b5dd93e6/build.openshift.io/builds.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183772135Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.183845886Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/configmaps.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18398277Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/endpoints.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.184084912Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/events.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.184408881Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/persistentvolumeclaims.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.184506023Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/pods.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.184599186Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/replicationcontrollers.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18475754Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/secrets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.184946025Z namespaces/e2e-test-llm-inference-service-b5dd93e6/core/services.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185024737Z namespaces/e2e-test-llm-inference-service-b5dd93e6/discovery.k8s.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185071588Z namespaces/e2e-test-llm-inference-service-b5dd93e6/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18514521Z namespaces/e2e-test-llm-inference-service-b5dd93e6/image.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185190751Z namespaces/e2e-test-llm-inference-service-b5dd93e6/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185257433Z namespaces/e2e-test-llm-inference-service-b5dd93e6/k8s.ovn.org/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185304904Z namespaces/e2e-test-llm-inference-service-b5dd93e6/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185406147Z namespaces/e2e-test-llm-inference-service-b5dd93e6/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185454658Z namespaces/e2e-test-llm-inference-service-b5dd93e6/monitoring.coreos.com/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185518409Z namespaces/e2e-test-llm-inference-service-b5dd93e6/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185565971Z namespaces/e2e-test-llm-inference-service-b5dd93e6/networking.k8s.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185631032Z namespaces/e2e-test-llm-inference-service-b5dd93e6/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185679604Z namespaces/e2e-test-llm-inference-service-b5dd93e6/policy/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185751856Z namespaces/e2e-test-llm-inference-service-b5dd93e6/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185805387Z namespaces/e2e-test-llm-inference-service-b5dd93e6/route.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185873119Z namespaces/e2e-test-llm-inference-service-b5dd93e6/route.openshift.io/routes.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18592026Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.185989282Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186053114Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186113585Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186160936Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186224958Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18631922Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186415603Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186507145Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186567527Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186615858Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18667693Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186728641Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186839304Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186886875Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.186948617Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.187044009Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.187107361Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.187154982Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.187281555Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.187371777Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18747154Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.187565033Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.187663015Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.187814209Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188005554Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188055685Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188122567Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188169068Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.1882326Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188279511Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188346623Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188440695Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188499267Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188546408Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18860803Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188656751Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188728973Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188776474Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188846036Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188884467Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.188958669Z namespaces/kuadrant-system/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.18900451Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.189078402Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.189124153Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.189188525Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.189233766Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.189334279Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.189843032Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.19019089Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190240592Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190298803Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190345485Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190409966Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190503469Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.19056354Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190611752Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190723284Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190788946Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190832617Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.190989641Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.191114274Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.191600427Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.191674429Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.19209988Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.192223263Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.192458229Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.192612123Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.192652384Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.192817538Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.19286323Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.192927712Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.192964162Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193069805Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193122037Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193174718Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193234389Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193282711Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193348452Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193366533Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193393933Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino-7cf4765b85-ft7f4.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193498316Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193507127Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193512076Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.193530797Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.22184524Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.221876191Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.221930722Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.221973373Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/authorino-operator-56fb6f6856-nq65w.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222082976Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222092056Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222097206Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222110227Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222328022Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222413435Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222464366Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222521477Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/dns-operator-controller-manager-ccbdf686-hsv2h.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.22262942Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.22263656Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.22264081Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222678871Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222820195Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222904677Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.222955089Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.22300005Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin-598f86468b-ffwbm.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223095462Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223108922Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223114053Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223150214Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223242836Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223332988Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.2233781Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223436521Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/kuadrant-operator-controller-manager-66866f5fcc-7phb4.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223560244Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223567354Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223571354Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.223621136Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242056143Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242122715Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242169466Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242243758Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador-limitador-85d9db4d5d-glmwj.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.24234658Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242375281Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242382791Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242394682Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242527385Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242605127Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242636078Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242692719Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/limitador-operator-controller-manager-774d94476f-l7cg9.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242823103Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242834933Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242846233Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.242856534Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243028328Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.24309479Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243133681Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243160461Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243220363Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243270104Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243335826Z namespaces/openshift-ingress/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243385947Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.2434792Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243504861Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243585943Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243630894Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.243752727Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244009074Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244215019Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.24427419Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244322302Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244400904Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244447645Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244551277Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244600189Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.24466264Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244778074Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244841725Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.244889386Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.245322848Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.245453621Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.245680347Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.24580504Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246066367Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246221891Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246427616Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246523199Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.24657869Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246680332Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246741204Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246808456Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246841457Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.24695418Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.246996171Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247038452Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247092283Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247131004Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247188406Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247196566Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247246627Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/istiod-openshift-gateway-5677ccb49b-x4v6g.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.24736226Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247372941Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247377601Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.247431722Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.269098173Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.269157944Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.269197055Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.269276028Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.269407181Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.269416211Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.269421351Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.269468622Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272319196Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272420839Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.27247167Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router-default-5dc8cf7cff-gn2tw.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272595823Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272604454Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272615594Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272651575Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/current.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272796919Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/previous.insecure.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272882281Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/previous.log [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272931222Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.272984203Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.273034435Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.273107247Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.281060242Z [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.281075293Z sent 3,413 bytes received 1,433,270 bytes 957,788.67 bytes/sec [must-gather] [must-gather-pjdrd] OUT 2026-07-28T18:56:52.281102534Z total size is 25,016,778 speedup is 17.41 [must-gather] [must-gather ] OUT 2026-07-28T18:56:52.336943439Z namespace/openshift-must-gather-867wq deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: ae3af18b-c19b-4ff5-9c6e-bc65d4c0a656 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T18:56:52.444902693Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: ae3af18b-c19b-4ff5-9c6e-bc65d4c0a656 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T18:56:52.463260198Z namespace/openshift-must-gather-67kr8 created [must-gather] [must-gather ] OUT 2026-07-28T18:56:52.470451924Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-p4q7z created [must-gather] [must-gather ] OUT 2026-07-28T18:56:52.488658775Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:53.727018697Z [disk usage checker] Started [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:53.730395915Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:53.942172020Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:54.091668414Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:54.190109049Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:54.282072217Z Gathering data for ns/default... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:54.627775457Z Gathering data for ns/openshift... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:55.089418408Z Gathering data for ns/kube-system... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:55.881670593Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.219640553Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.219737905Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.219737905Z namespaces "assisted-installer" not found [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.808922772Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.811937990Z INFO: Collecting host service logs for crio [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.812196277Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.812528347Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.812845488Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.813171248Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.813567872Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.813836869Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.814157688Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.814546930Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.814897868Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.815226368Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.815651416Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.822683333Z INFO: Gathering HAProxy config files [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.827659575Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.835458536Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.835642590Z Executing Istio gather script [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.836570984Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:56.855018253Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:57.654863477Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:57.843835389Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:57.951382016Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:57.965979591Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.014408557Z No resources found [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.035908165Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.275042661Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.449011533Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-zxbld [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.449059767Z insights-runtime-extractor-6ffmn [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.449080088Z insights-runtime-extractor-8dpn4 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.449099716Z insights-runtime-extractor-fg6ls [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.694465899Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.694594076Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.812122877Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.891062514Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:58.970933276Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.004799754Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.051949701Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.067811154Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.091357952Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.177386588Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.445123731Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.607304151Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.621247211Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.677243624Z INFO: 'previous-logs' folder not found on ip-10-0-133-215.ec2.internal, skipping... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.695159008Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.731130641Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.731130641Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.804402689Z tar: Removing leading `/' from member names [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.811296040Z INFO: HAProxy config collection complete. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:56:59.872077886Z No resources found [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.049964757Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.194361773Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.211487259Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.275164753Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.373798054Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.464789642Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.539660732Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.548025570Z INFO: 'previous-logs' folder not found on ip-10-0-136-111.ec2.internal, skipping... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:00.900163415Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.054251932Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.059481065Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.076777938Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.318658182Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.339660858Z INFO: 'previous-logs' folder not found on ip-10-0-137-238.ec2.internal, skipping... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.339780078Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.339830799Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.410626982Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.448540379Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.598942965Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.603373716Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.616670449Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.820400077Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.827896218Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:01.980226207Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.065800186Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.069674893Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.213540824Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.232740586Z INFO: INTERCONNECT MODE [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.232839080Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.337774926Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.387678219Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.391376101Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.493063085Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.498745345Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.504574120Z INFO: Gathering OVN_Northbound from ovnkube-node-b92c2... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.505360198Z INFO: Gathering OVN_Northbound from ovnkube-node-scdr5... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.505360198Z INFO: Gathering OVN_Northbound from ovnkube-node-v552f... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.505360198Z INFO: Gathering OVN_Southbound from ovnkube-node-b92c2... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.506852153Z INFO: Gathering OVN_Southbound from ovnkube-node-scdr5... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.506852153Z INFO: Gathering OVN_Southbound from ovnkube-node-v552f... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.804993015Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.865798979Z tar: Removing leading `/' from member names [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.865898003Z tar: Removing leading `/' from member names [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.930146467Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.931500609Z tar: Removing leading `/' from member names [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.960173992Z tar: Removing leading `/' from member names [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:02.960456036Z tar: Removing leading `/' from member names [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.009026833Z tar: Removing leading `/' from member names [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.039048770Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.043680410Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.195903757Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.332121289Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.365037777Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.701630068Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.701768427Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.702336630Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.707597886Z 8.3M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.714289778Z ovnk_database_store/ [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.714289778Z ovnk_database_store/ovnkube-node-b92c2_nbdb [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.742661528Z ovnk_database_store/ovnkube-node-v552f_sbdb [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.749298892Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.751921934Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.806787143Z ovnk_database_store/ovnkube-node-b92c2_sbdb [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.876584002Z ovnk_database_store/ovnkube-node-scdr5_sbdb [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.927168340Z ovnk_database_store/ovnkube-node-scdr5_nbdb [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.944073323Z ovnk_database_store/ovnkube-node-v552f_nbdb [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.964179061Z INFO: Network log collection complete. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:03.994083708Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:04.092127038Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:04.411827401Z W0728 18:57:04.411763 1617 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:04.419387270Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:04.428913911Z W0728 18:57:04.428863 322 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:04.441865821Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:04.689964805Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:04.777780996Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.099705312Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.103092623Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.200405845Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.639843139Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.649824562Z Gathering data for ns/openshift-console... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.652708314Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.817090633Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.891100360Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:05.985769981Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:06.144628860Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:06.147257434Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:06.322526976Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:06.467031982Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:06.715091855Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:06.717982426Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:06.935377097Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:06.943362837Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.149267258Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.248583550Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.252275962Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.308142125Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.440856651Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.672625667Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.675001630Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.914802303Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:07.982365490Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.157395499Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.160748602Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.309172485Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.382487534Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.564316764Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.628086319Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.631447649Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.844642866Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:08.963929786Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.000413821Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.092237947Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.096370152Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.361909729Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.473291002Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.615628658Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.619283148Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.811666902Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:09.949842253Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.040423894Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.045056101Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.258019493Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.408201392Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.498347984Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.501571506Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.626463747Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.705222103Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.942642682Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:10.945097570Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:11.163919429Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:11.403125044Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:11.405987912Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:11.594792220Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:11.774289909Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:11.825337686Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:11.828893683Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:12.015750282Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:12.231956211Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:12.235478925Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:12.436943330Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:12.649636624Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:12.651979466Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:12.866132700Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:12.929718698Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:13.105379329Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:13.108639725Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:13.314996621Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:13.546905339Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:13.549442904Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:13.566992409Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:13.788987474Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.008576305Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.034442484Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.038247080Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.116497110Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.267127294Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.270695710Z Gathering data for ns/kserve... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.452635674Z Collecting performance related data for node ip-10-0-136-111.ec2.internal [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.461697306Z Collecting performance related data for node ip-10-0-133-215.ec2.internal [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.470248648Z Collecting performance related data for node ip-10-0-137-238.ec2.internal [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.520589210Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.523998514Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:14.748667451Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:15.003700578Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:15.007230900Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:15.244627515Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:15.513265301Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:15.671913583Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:15.938933071Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:16.664376539Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:16.825258757Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:17.008235050Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:17.165459240Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:17.455435347Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:18.130998112Z Gathering data for ns/openshift-config... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:18.189136779Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:18.375729129Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:18.464186704Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:18.609062120Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:19.015258223Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:19.023430061Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:19.143515460Z Done executing Istio gather script [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:19.364419751Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:19.709872155Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:20.098130510Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:20.473721292Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:20.789009847Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:21.159089646Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:21.535899042Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:21.896992270Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:22.517869090Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:23.391107281Z Collecting kubelet logs for node ip-10-0-136-111.ec2.internal [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:23.393205740Z Collecting kubelet logs for node ip-10-0-133-215.ec2.internal [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:23.395473114Z Collecting kubelet logs for node ip-10-0-137-238.ec2.internal [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:23.849022042Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-67kr8 namespace [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:23.856902520Z INFO: Node performance data collection complete. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:24.020274886Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:24.137951095Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:25.532048954Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:25.870047214Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:26.437212488Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:26.908724319Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:27.314102797Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:27.607291661Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:28.116697321Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:28.441655058Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:28.905414689Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:29.026955676Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:29.224297075Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:29.549213259Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:30.262629523Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:30.597849874Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:30.935540122Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:31.346773331Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:32.329660749Z Wrote inspect data to must-gather. [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:32.329712388Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:32.329712388Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:32.484956483Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-k5rkf] POD 2026-07-28T18:57:32.724309595Z Caches written to disk [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:33.734646003Z waiting for gather to complete [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:33.737059113Z downloading gather output [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:33.977762456Z receiving incremental file list [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:33.998305643Z ./ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:33.998433566Z aggregated-discovery-api.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:33.998815966Z aggregated-discovery-apis.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:33.999806081Z event-filter.html [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.002741555Z timestamp [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.002841147Z version [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.009564086Z cluster-scoped-resources/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.009583897Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.009600117Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.009683229Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.009870574Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.009995897Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.010062849Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.01010856Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.010221503Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.010320935Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.010446698Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.010471919Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.010641113Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.010725735Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.011160106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.011186317Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.011416503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.011683799Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.011904195Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.012064669Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.015954737Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.017814273Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.017907216Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.018022149Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.018154182Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.018280055Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.018774648Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.018911741Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.019087405Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.019201268Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.020024899Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.020260385Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.020378218Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.020551652Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.021728192Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.021980238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.022147162Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.022300286Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.022619825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.022773688Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.022972923Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.02322913Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.024001149Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.024117552Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.024232635Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.024347988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.02444529Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.024550943Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.024735407Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.024920802Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.025692472Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.025852626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.025974759Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.027865016Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.028175204Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.028280327Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.02842555Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.028735528Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.029169689Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.029282462Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.029404295Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.029518818Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.029625021Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.029748984Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.029877917Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.03002176Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.030157814Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.030261977Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.030563074Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.030697528Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.030823271Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.031349254Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.031467847Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.0315841Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.032737989Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.033627831Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.033801926Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.034054952Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.034177885Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.03436947Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.034496563Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.034665527Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.034825021Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.034959515Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.035080678Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.035231422Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.035365765Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.035472528Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.035629201Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.035807896Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.035940379Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.036096963Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.036517884Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.036691958Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.040122005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.040254848Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.041219082Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.041342475Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.041466058Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.041580391Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.04191157Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.042035323Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.042242238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.042371691Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.042485804Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.042661768Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.042819692Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.042941815Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.043095979Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.043223283Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.047341896Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.047788167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.0483103Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.048504945Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.048645469Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.048806903Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.048939846Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.049047369Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.049147861Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.049964002Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.050252029Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.055778728Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.055897101Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.056035105Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.056159138Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.056280071Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.056391924Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.056506126Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.059075521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.062937848Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.068020626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.073185046Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.07334122Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.073618947Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.073804601Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.073948835Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.074068298Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.074207092Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.074466938Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.074574931Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.074678514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.074892499Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.075028612Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.075140815Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.075256678Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.075368731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.075519455Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.075688529Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.075834103Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.075953146Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.076076449Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.076242873Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.076346625Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.076481269Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.076590871Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.076727125Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.077014252Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.077140155Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.077379771Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.077491804Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.077606217Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.07772402Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.079740901Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.079891714Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.079981237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.080071169Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.080178352Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.080396027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.080437818Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.080589602Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.080733766Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.082328596Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.08249165Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.082634174Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.082820998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.082929461Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.083163637Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.083531736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.083949207Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.084214653Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.084332576Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.084442369Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.084552532Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.085159507Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.085369912Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.085482465Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.086927782Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.087062015Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.087207649Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.087312721Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.087461375Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.087764563Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.087886576Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.08804223Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.088426969Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.088525802Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.088637285Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.089688791Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.089919937Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.090065241Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.090208224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.090358828Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.090668626Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.090735547Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.09083724Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.090915652Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091004564Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091073366Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091219239Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091318622Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091403134Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091490696Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091574988Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091669781Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091770093Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091860135Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.091950478Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.09204233Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092129072Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092216004Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092301237Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092391039Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092473681Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092564423Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092645455Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092751058Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.092934823Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093019275Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093106017Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093181549Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093284471Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093368853Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093455886Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093535518Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.09361888Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093729693Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093833725Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.093922158Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094007249Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094103462Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094209595Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094361609Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094373949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094441811Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094527183Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094616055Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094705097Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.09480904Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094895552Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.094986564Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095103827Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095190739Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095292702Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095383074Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095473766Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095565859Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095657761Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095770994Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095856516Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.095949408Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096047841Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096138673Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096238276Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096323838Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.09642287Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096506292Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096602645Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096736678Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096824511Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.096920163Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097005935Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097098497Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.0971854Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097275572Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097372194Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097456196Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097543718Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097638701Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097728053Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.097859566Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.09797986Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098066292Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098152814Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098249056Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098333569Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098431741Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098523333Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098620086Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098724098Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098846451Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.098940124Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.099027166Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.099113468Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.09921028Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.099326633Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.099417486Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.099503568Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.09959695Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.099692103Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.099814566Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.099919428Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10000612Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100104103Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100194685Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100287438Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10037828Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100470292Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100560764Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100654417Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100722749Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100742489Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100812141Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-79z2f.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.100943844Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-j5f95.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.101034946Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-zmfxh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10118366Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.101224741Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.101385735Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.101521929Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.101611951Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.101695363Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102129044Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102236496Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102325279Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102418611Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102534814Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102620896Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102720049Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102820561Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.102906773Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103002736Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103126759Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10319056Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103290363Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103371405Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103454347Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103545049Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103630082Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103737244Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103866868Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.103907709Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.104026932Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.104107634Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.104198736Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.104285888Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10437768Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.104481883Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.104631567Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.104730259Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105017257Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105106729Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105202441Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105341155Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105439177Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105529379Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105618292Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105737175Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.105852118Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10594382Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106038362Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106138305Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106203556Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106251748Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106313699Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10635485Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106400831Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106442822Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106509044Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106551585Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106610177Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106640617Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106688448Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10675787Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106810612Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106821582Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106844263Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.106948055Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107020747Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107072108Z cluster-scoped-resources/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107083209Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10714761Z cluster-scoped-resources/core/nodes/ip-10-0-133-215.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107349975Z cluster-scoped-resources/core/nodes/ip-10-0-136-111.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107490049Z cluster-scoped-resources/core/nodes/ip-10-0-137-238.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107588221Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107657703Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107722354Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107822687Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107906629Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.107990101Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.108074553Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.108202377Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10834524Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.108475853Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.108597276Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.108690229Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.108860853Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.108964486Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109048498Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10914761Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109242253Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109331855Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109426567Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109508209Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109598532Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109695574Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109822927Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.10991014Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.109959321Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110053053Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110130715Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110216547Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110299529Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110389181Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110545085Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110673819Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110794162Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110851463Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110859173Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.110914785Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111000527Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111010117Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111046268Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11113053Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111160461Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111225453Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111239903Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111274914Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111376066Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11152447Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111599422Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111611582Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111644113Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111764156Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111775856Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111812077Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.111894169Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11193307Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112055533Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112094784Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112172186Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112206467Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112281939Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11232283Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112444473Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112486014Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112550556Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112595127Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112663559Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112691669Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112782972Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112805082Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112882804Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112924485Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.112987387Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113028728Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113062429Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11311739Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113182072Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113227703Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113282184Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113327746Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113426028Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113433878Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113460209Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11351792Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113558251Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113667014Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113801578Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.113998872Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114110515Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114221668Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114329521Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114401922Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114466344Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114493245Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114598178Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11468256Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114852204Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.114982487Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115072699Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115170702Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115257724Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115347426Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115427248Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11550528Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115597652Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115698045Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115803328Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.1158887Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.115983472Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.116067814Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.116150146Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.116391823Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.116468545Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.116560207Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.116726991Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.116860564Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.116963517Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117044579Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117132011Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117276455Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117364657Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117456109Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117542161Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117630744Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117748707Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117839059Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.117937321Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118020204Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118105775Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118190248Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11828917Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118385093Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118471475Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118562557Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118654079Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118765292Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118863135Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.118950127Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119046779Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119130981Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119221934Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119314076Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119407158Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11949052Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119583653Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119636034Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119645654Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119677435Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119752007Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119764717Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119800738Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11985824Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.11987194Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119897631Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119947622Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.119995143Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-133-215.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120087895Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-136-111.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120173478Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-137-238.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120241909Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.12028344Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120373772Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120427164Z host_service_logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120443554Z host_service_logs/masters/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120496246Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120570828Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120635959Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120722541Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120859455Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.120972848Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121043889Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121110501Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121183573Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121247205Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121308656Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121349677Z ingress_controllers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121359847Z ingress_controllers/default/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121364808Z ingress_controllers/default/router-default-5dc8cf7cff-gn2tw/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121407249Z ingress_controllers/default/router-default-5dc8cf7cff-gn2tw/haproxy.config [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121529142Z insights-data/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121537792Z istio/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121569473Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.121686896Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.12226992Z istio/event-filter.html [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122420724Z istio/timestamp [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122551647Z istio/cluster-scoped-resources/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122559898Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122564398Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122614569Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122703731Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122779003Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122845905Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.122949188Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.123012319Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.123188123Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.124198469Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.124410764Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.124589079Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.12505558Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.125229595Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.125628355Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.126481726Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.12664066Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.126769493Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.126904087Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.127007799Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.127121602Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.127268866Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.12744629Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.127739648Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.127923893Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128309932Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128444726Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128576689Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128731243Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128749823Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128781424Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128849006Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128869516Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128908157Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.128995979Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129159133Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129233465Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129273456Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129370749Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129519953Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129586084Z istio/namespaces/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129596224Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129600875Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/gateway.networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129607475Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129661956Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/gateway.networking.k8s.io/httproutes/auth-invalid-token-test-kserve-route.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129788089Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/inference.networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.12980305Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129844151Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/inference.networking.k8s.io/inferencepools/auth-invalid-token-test-inference-pool.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129912752Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/inference.networking.x-k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129920483Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.129966284Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/inference.networking.x-k8s.io/inferencepools/auth-invalid-token-test-inference-pool.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130031115Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/networking.istio.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130045586Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/networking.istio.io/destinationrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130087707Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/networking.istio.io/destinationrules/auth-invalid-token-test-kserve-scheduler.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130180579Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/networking.istio.io/destinationrules/auth-invalid-token-test-kserve-shadow-svc.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130263911Z istio/namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/networking.istio.io/destinationrules/auth-invalid-token-test-kserve-workload-svc.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130417755Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130465597Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130532588Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130573819Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130642841Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130686132Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.130782745Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131048371Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131252696Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131306628Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131343079Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.13142435Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131469352Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131558824Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131616275Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131649016Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131832181Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131919243Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.131975084Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.132383665Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.132486177Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.132721533Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.132811945Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133182795Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133309288Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133513503Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133613096Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133650757Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133728848Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133741369Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.13380021Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133889592Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133924143Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.133984715Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134020286Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134107578Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.1341864Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134235941Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134271642Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134329563Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134342874Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134384805Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134471647Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.13459667Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134683083Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134736854Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134799385Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134809706Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134844527Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/istiod-openshift-gateway-5677ccb49b-x4v6g.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134943369Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.134951729Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.13496443Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.135018201Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.146770306Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.146845618Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.146871199Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.1469194Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.147034853Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.147043783Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.147061824Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.147087394Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149017953Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149125436Z istio/namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149169517Z istio/namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router-default-5dc8cf7cff-gn2tw.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.14929295Z istio/namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.14930272Z istio/namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.14931658Z istio/namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149366112Z istio/namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149477884Z istio/namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149550256Z istio/namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149567247Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149625128Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149677929Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.14969118Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149788762Z machine_config_ondisk/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149803233Z machine_config_termination_logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149807973Z monitoring/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149819793Z monitoring/alertmanager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149828053Z monitoring/alertmanager/status.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149928166Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.149996068Z monitoring/prometheus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.150016218Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.150123491Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.150183102Z monitoring/prometheus/rules.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.150853019Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.150906211Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.150943241Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.151875855Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.151925366Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.151971657Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.152054009Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.152124391Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.152215293Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.152273245Z monitoring/prometheus/status/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.152320456Z monitoring/prometheus/status/config.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.152792108Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.152853889Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.152953342Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153152197Z namespaces/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153172677Z namespaces/cert-manager-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153180578Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153187618Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153233979Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153313021Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153324421Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153388933Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153573047Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153818064Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.153968227Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.154191723Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.154423799Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.15446939Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-8kq6k.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.154632754Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.154677205Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.154758137Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.154799738Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.15486307Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.15488512Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.154977213Z namespaces/cert-manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.154989353Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.155003974Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.155045824Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.155197848Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.155372783Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.155505286Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.155702371Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.155946607Z namespaces/default/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.155997879Z namespaces/default/default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.15604656Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156096401Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156164643Z namespaces/default/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156206094Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156287246Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156359038Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.15644426Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156494171Z namespaces/default/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156525112Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156571223Z namespaces/default/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156617094Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156703716Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156774888Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156817819Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156904271Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.156982233Z namespaces/default/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157028374Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157134127Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157216879Z namespaces/default/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157388714Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157464125Z namespaces/default/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157547327Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157675531Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157877326Z namespaces/default/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157945928Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.157983049Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.15804465Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158079481Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158127572Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158160773Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158249375Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158325857Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158380308Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158411369Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.15845677Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158504591Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158563573Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158574653Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158623805Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158806679Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.158980574Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159116687Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159316272Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159511537Z namespaces/default/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159556308Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159596499Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.15964303Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159700702Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159725992Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159740823Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159803314Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159897017Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159955278Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.159999729Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.160083271Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.160218285Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.160228535Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.160290996Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16044273Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.160617215Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.160805469Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.161012974Z namespaces/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.161211599Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16122263Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16123152Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.161279371Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.161433945Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.1616139Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.161767953Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.161994719Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.162190244Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.162202015Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.162209895Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.162262906Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16242373Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.162607285Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.162760089Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.162965074Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.163148648Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.163160568Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.163165669Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.163173079Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.163339473Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.163522848Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.163658361Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.163888827Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.164067171Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.164075882Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.164093072Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.164134573Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.164285297Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.164465801Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.164584675Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16481306Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.164995525Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.165004065Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.165015975Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.165057686Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16520945Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.165385995Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.165530328Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.165830096Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166053231Z namespaces/kserve/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166086432Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166169004Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166194815Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166280387Z namespaces/kserve/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166310538Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16639572Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166633826Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166865282Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166913093Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.166932774Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167014005Z namespaces/kserve/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167045076Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167134399Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167227171Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167259572Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167353084Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167384045Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167388655Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167440256Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167529758Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167614701Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167731454Z namespaces/kserve/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.167779565Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.168014121Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.168124304Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.168413301Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.168494573Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.168821971Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.168947074Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169220251Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169318123Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169363555Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169458937Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169488338Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16956615Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.16959653Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169694833Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169819566Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169864527Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.169913908Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.1699829Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170032081Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170118654Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170184225Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170236506Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170283898Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170320649Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.17038024Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170607736Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170809171Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.170950435Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171150809Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171333474Z namespaces/kserve/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171350014Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-d9v9x/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171403696Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-d9v9x/kserve-controller-manager-cd4d44c67-d9v9x.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171520619Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-d9v9x/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171531229Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-d9v9x/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171539219Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-d9v9x/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.17157916Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-d9v9x/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171705574Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-d9v9x/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171813116Z namespaces/kserve/pods/kserve-controller-manager-cd4d44c67-d9v9x/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171876708Z namespaces/kserve/pods/llmisvc-controller-manager-84fb8458f8-4rlxl/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.171922919Z namespaces/kserve/pods/llmisvc-controller-manager-84fb8458f8-4rlxl/llmisvc-controller-manager-84fb8458f8-4rlxl.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.172013121Z namespaces/kserve/pods/llmisvc-controller-manager-84fb8458f8-4rlxl/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.172030692Z namespaces/kserve/pods/llmisvc-controller-manager-84fb8458f8-4rlxl/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.172039042Z namespaces/kserve/pods/llmisvc-controller-manager-84fb8458f8-4rlxl/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.172087013Z namespaces/kserve/pods/llmisvc-controller-manager-84fb8458f8-4rlxl/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.192874556Z namespaces/kserve/pods/llmisvc-controller-manager-84fb8458f8-4rlxl/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.192925407Z namespaces/kserve/pods/llmisvc-controller-manager-84fb8458f8-4rlxl/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.192993089Z namespaces/kserve/pods/model-serving-api-7b954ffc95-w4pv6/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.19302972Z namespaces/kserve/pods/model-serving-api-7b954ffc95-w4pv6/model-serving-api-7b954ffc95-w4pv6.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193135613Z namespaces/kserve/pods/model-serving-api-7b954ffc95-w4pv6/server/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193147153Z namespaces/kserve/pods/model-serving-api-7b954ffc95-w4pv6/server/server/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193161543Z namespaces/kserve/pods/model-serving-api-7b954ffc95-w4pv6/server/server/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193199444Z namespaces/kserve/pods/model-serving-api-7b954ffc95-w4pv6/server/server/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193288316Z namespaces/kserve/pods/model-serving-api-7b954ffc95-w4pv6/server/server/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193365758Z namespaces/kserve/pods/model-serving-api-7b954ffc95-w4pv6/server/server/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.19341836Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zhv7r/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.19344155Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zhv7r/odh-model-controller-75d9cdbddf-zhv7r.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193552973Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zhv7r/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193565863Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zhv7r/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193572833Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zhv7r/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.193615194Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zhv7r/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.23717691Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zhv7r/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237247922Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-zhv7r/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237312393Z namespaces/kserve/pods/s3-init-qf8qs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237358954Z namespaces/kserve/pods/s3-init-qf8qs/s3-init-qf8qs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.23757075Z namespaces/kserve/pods/s3-init-qf8qs/download-hf-model/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237609611Z namespaces/kserve/pods/s3-init-qf8qs/download-hf-model/download-hf-model/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237617441Z namespaces/kserve/pods/s3-init-qf8qs/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237629791Z namespaces/kserve/pods/s3-init-qf8qs/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237752404Z namespaces/kserve/pods/s3-init-qf8qs/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237854427Z namespaces/kserve/pods/s3-init-qf8qs/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237876987Z namespaces/kserve/pods/s3-init-qf8qs/s3-init/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237881198Z namespaces/kserve/pods/s3-init-qf8qs/s3-init/s3-init/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.237887978Z namespaces/kserve/pods/s3-init-qf8qs/s3-init/s3-init/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.23796584Z namespaces/kserve/pods/s3-init-qf8qs/s3-init/s3-init/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238302348Z namespaces/kserve/pods/s3-init-qf8qs/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.23838323Z namespaces/kserve/pods/s3-init-qf8qs/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238423901Z namespaces/kserve/pods/seaweedfs-64568bcd49-g7wgt/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238480863Z namespaces/kserve/pods/seaweedfs-64568bcd49-g7wgt/seaweedfs-64568bcd49-g7wgt.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238569565Z namespaces/kserve/pods/seaweedfs-64568bcd49-g7wgt/seaweedfs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238577025Z namespaces/kserve/pods/seaweedfs-64568bcd49-g7wgt/seaweedfs/seaweedfs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238602816Z namespaces/kserve/pods/seaweedfs-64568bcd49-g7wgt/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238644567Z namespaces/kserve/pods/seaweedfs-64568bcd49-g7wgt/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238899243Z namespaces/kserve/pods/seaweedfs-64568bcd49-g7wgt/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.238978825Z namespaces/kserve/pods/seaweedfs-64568bcd49-g7wgt/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239031687Z namespaces/kserve/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239066288Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239135789Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.23918211Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239292003Z namespaces/kuadrant-system/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239359935Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239445807Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239486898Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.23957367Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239614981Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.239726414Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240043932Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240319589Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240375641Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240437702Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240509534Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240565215Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240663198Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240722959Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240796021Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240890503Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240948985Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.240994586Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.241045867Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.24115251Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.241252192Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.241355795Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.241410047Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.241472708Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.241624742Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.241791476Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.242240127Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.242317999Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.242647718Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.242844612Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243092169Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243213112Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243259883Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243382976Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243429237Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243497389Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.24354225Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243645253Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243760226Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243806387Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243874958Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.24392147Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.243979321Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.244058633Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.244071533Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.244142465Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.244359721Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.244583966Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.244759131Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.244985196Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.245199142Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.245247953Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-d85dq.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.245617462Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.245652603Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.245781086Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.245873428Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.245954301Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246005832Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246036633Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246101924Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246150075Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246246788Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246398122Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246493904Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246573256Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246580426Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246627227Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino-7cf4765b85-ft7f4.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.24673876Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246754301Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246770131Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.246822263Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.283536246Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.283577917Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-ft7f4/authorino/authorino/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.283614188Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.28369509Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/authorino-operator-56fb6f6856-nq65w.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.283823753Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.283843543Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.283854904Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.283910525Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.28412059Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284186832Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-nq65w/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284233463Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284287945Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/dns-operator-controller-manager-ccbdf686-hsv2h.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284384827Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284404458Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284415818Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284450419Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284555061Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284629583Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-hsv2h/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284680195Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284742826Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin-598f86468b-ffwbm.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284836068Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284849209Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284856959Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.28490354Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.284989622Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.285057554Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-ffwbm/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.285076174Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.285134786Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/kuadrant-operator-controller-manager-66866f5fcc-7phb4.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.285242349Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.285252929Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.285278279Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.28528779Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.306746599Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.306801881Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-7phb4/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.306830281Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.306893763Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador-limitador-85d9db4d5d-glmwj.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.306983075Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.306992436Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.306997106Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307034787Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307126149Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307197681Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-glmwj/limitador/limitador/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307252112Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307289923Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/limitador-operator-controller-manager-774d94476f-l7cg9.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307429487Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307441757Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307446077Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307482928Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307656562Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307734824Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-l7cg9/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307793936Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307852167Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.307875578Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.30794614Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308000951Z namespaces/kube-node-lease/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308015931Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308023131Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308061583Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-133-215.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308157475Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-136-111.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308240377Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-137-238.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308291008Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308301478Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.30836864Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308557175Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308791651Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.308942575Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.309187251Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.309441857Z namespaces/kube-public/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.309451078Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.309458068Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.309525189Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.309690964Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.30993701Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.310098834Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.310319509Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.310576696Z namespaces/kube-system/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.310611247Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.310687149Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3107462Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.310824442Z namespaces/kube-system/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.310889114Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311043518Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.31113576Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311218882Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311273793Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311320454Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311400907Z namespaces/kube-system/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311420407Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311512339Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.31154319Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311594411Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311670023Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311737135Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311757545Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311790096Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-c5l4d3ta4pd7wh3zkg5rfsal5q.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311879388Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.311966911Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.312052493Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.312223597Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.312290919Z namespaces/kube-system/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.31232821Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.312482144Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.312566066Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.312847933Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.312925125Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313223142Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313352005Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313660243Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313729685Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313791887Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313850828Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313891039Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313955331Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.313996552Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314078864Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314174146Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314229507Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3143169Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314380151Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314438163Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314508245Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314516635Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314589657Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314807212Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.314997797Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.31514121Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.315355826Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.315665384Z namespaces/kube-system/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.315678224Z namespaces/kube-system/pods/global-pull-secret-syncer-qf77t/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.315734405Z namespaces/kube-system/pods/global-pull-secret-syncer-qf77t/global-pull-secret-syncer-qf77t.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.315821578Z namespaces/kube-system/pods/global-pull-secret-syncer-qf77t/global-pull-secret-syncer/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.315831718Z namespaces/kube-system/pods/global-pull-secret-syncer-qf77t/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.315845618Z namespaces/kube-system/pods/global-pull-secret-syncer-qf77t/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.315884669Z namespaces/kube-system/pods/global-pull-secret-syncer-qf77t/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.31631019Z namespaces/kube-system/pods/global-pull-secret-syncer-qf77t/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.316411242Z namespaces/kube-system/pods/global-pull-secret-syncer-qf77t/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.316489255Z namespaces/kube-system/pods/global-pull-secret-syncer-vcff5/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.316545006Z namespaces/kube-system/pods/global-pull-secret-syncer-vcff5/global-pull-secret-syncer-vcff5.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.316623018Z namespaces/kube-system/pods/global-pull-secret-syncer-vcff5/global-pull-secret-syncer/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.316630778Z namespaces/kube-system/pods/global-pull-secret-syncer-vcff5/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.316635078Z namespaces/kube-system/pods/global-pull-secret-syncer-vcff5/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.316677349Z namespaces/kube-system/pods/global-pull-secret-syncer-vcff5/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317137051Z namespaces/kube-system/pods/global-pull-secret-syncer-vcff5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317256674Z namespaces/kube-system/pods/global-pull-secret-syncer-vcff5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317285664Z namespaces/kube-system/pods/global-pull-secret-syncer-x8kpv/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317322016Z namespaces/kube-system/pods/global-pull-secret-syncer-x8kpv/global-pull-secret-syncer-x8kpv.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317411268Z namespaces/kube-system/pods/global-pull-secret-syncer-x8kpv/global-pull-secret-syncer/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317419918Z namespaces/kube-system/pods/global-pull-secret-syncer-x8kpv/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317425208Z namespaces/kube-system/pods/global-pull-secret-syncer-x8kpv/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317473309Z namespaces/kube-system/pods/global-pull-secret-syncer-x8kpv/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.31787895Z namespaces/kube-system/pods/global-pull-secret-syncer-x8kpv/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317955061Z namespaces/kube-system/pods/global-pull-secret-syncer-x8kpv/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.317975102Z namespaces/kube-system/pods/konnectivity-agent-bfd75/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.318039754Z namespaces/kube-system/pods/konnectivity-agent-bfd75/konnectivity-agent-bfd75.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.318130966Z namespaces/kube-system/pods/konnectivity-agent-bfd75/konnectivity-agent/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.318140406Z namespaces/kube-system/pods/konnectivity-agent-bfd75/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.318158286Z namespaces/kube-system/pods/konnectivity-agent-bfd75/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.318213478Z namespaces/kube-system/pods/konnectivity-agent-bfd75/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.319686485Z namespaces/kube-system/pods/konnectivity-agent-bfd75/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.319789718Z namespaces/kube-system/pods/konnectivity-agent-bfd75/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.319842619Z namespaces/kube-system/pods/konnectivity-agent-f8nqb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.31988774Z namespaces/kube-system/pods/konnectivity-agent-f8nqb/konnectivity-agent-f8nqb.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.319981622Z namespaces/kube-system/pods/konnectivity-agent-f8nqb/konnectivity-agent/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.319990903Z namespaces/kube-system/pods/konnectivity-agent-f8nqb/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.320003483Z namespaces/kube-system/pods/konnectivity-agent-f8nqb/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.320036404Z namespaces/kube-system/pods/konnectivity-agent-f8nqb/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.321158412Z namespaces/kube-system/pods/konnectivity-agent-f8nqb/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.321232184Z namespaces/kube-system/pods/konnectivity-agent-f8nqb/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.321295795Z namespaces/kube-system/pods/konnectivity-agent-l82pq/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.321322676Z namespaces/kube-system/pods/konnectivity-agent-l82pq/konnectivity-agent-l82pq.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.321405248Z namespaces/kube-system/pods/konnectivity-agent-l82pq/konnectivity-agent/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.321413208Z namespaces/kube-system/pods/konnectivity-agent-l82pq/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.321417838Z namespaces/kube-system/pods/konnectivity-agent-l82pq/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.321449629Z namespaces/kube-system/pods/konnectivity-agent-l82pq/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.32265875Z namespaces/kube-system/pods/konnectivity-agent-l82pq/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.322737361Z namespaces/kube-system/pods/konnectivity-agent-l82pq/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.322802573Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.322844814Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.322933006Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal/haproxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.322942787Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.322951257Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.322999988Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.323276185Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.323364327Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-215.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.323423089Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.32346727Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.323534942Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal/haproxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.323555232Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.323563852Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.323611394Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.324037074Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.324083255Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.324129947Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.324167188Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.32424792Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal/haproxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3242576Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.32426211Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.324312881Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325385858Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3254523Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325496301Z namespaces/kube-system/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325534722Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325580733Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325588273Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325624994Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325728367Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325885121Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325916812Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.325982723Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.326001314Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.326067245Z namespaces/observability/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.326079976Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.326095106Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.326131057Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.326314231Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.326550877Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.326730872Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.327011479Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.327240835Z namespaces/open-cluster-management-043e8889-7773-493c-bcb4-d7808116c/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.327256635Z namespaces/open-cluster-management-043e8889-7773-493c-bcb4-d7808116c/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.327267166Z namespaces/open-cluster-management-043e8889-7773-493c-bcb4-d7808116c/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.327324897Z namespaces/open-cluster-management-043e8889-7773-493c-bcb4-d7808116c/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.327506721Z namespaces/open-cluster-management-043e8889-7773-493c-bcb4-d7808116c/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.327729497Z namespaces/open-cluster-management-043e8889-7773-493c-bcb4-d7808116c/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.327907222Z namespaces/open-cluster-management-043e8889-7773-493c-bcb4-d7808116c/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.328525157Z namespaces/open-cluster-management-043e8889-7773-493c-bcb4-d7808116c/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.328807444Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.328824855Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.328831685Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.328846925Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.328933427Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.328973658Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.328996129Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.32903642Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.329219835Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.329396239Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.329539853Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.329769158Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330012935Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330063316Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330130658Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330175679Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.33025403Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330312742Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330388624Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330471416Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330554758Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.33061195Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330667991Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330734453Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330788944Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330870306Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330899207Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.330965779Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.331069491Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.331122062Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.331178764Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.340533179Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.340839097Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.340888678Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.340993491Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341066373Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341196676Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341393981Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341474953Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341497343Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341559175Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341608336Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341655957Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341734019Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341888563Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.341964575Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342008686Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342066118Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342129949Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34217601Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342232712Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342251792Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342329904Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342338725Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342362065Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342533899Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342725404Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.342861908Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343085373Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343277458Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343297539Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34334869Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343392121Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343493343Z namespaces/openshift-apiserver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343544165Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343605756Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343654958Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343686598Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34375896Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343840082Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343913764Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.343997996Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344285514Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344315304Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344387036Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344427687Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344511959Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34456275Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344605111Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344687804Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344785886Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.344821897Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34492673Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345008542Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345088064Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345167076Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345246748Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345374541Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345531295Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345579796Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345619347Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345686729Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34574905Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345836913Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345860033Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.345966996Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346042698Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346105459Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34613507Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346201542Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346245173Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346293064Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346303734Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346364466Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3465419Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346792446Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.346959341Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.347212707Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.347595567Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.347642888Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34774032Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.347803382Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.347859313Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.347871484Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.347876704Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.347946726Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.348028237Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.348039718Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.348106479Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.348290944Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34850646Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.348663213Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.34894356Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.349155046Z namespaces/openshift-authentication/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.349163486Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.349197107Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.349243648Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.349431013Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.349641088Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.349838203Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350072339Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350280944Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350288774Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350299865Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350352666Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350441598Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350457879Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35051795Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350699345Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.350928221Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351089125Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351369552Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351615458Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351671069Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35170546Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351798772Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351835063Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351859644Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351892675Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.351951526Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.352019768Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.352051019Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.352244874Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.352455719Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.352629783Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.352950841Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353200058Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35328038Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353359942Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353402193Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353461874Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353560697Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353659859Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353799543Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353935596Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.353955637Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354048549Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354070249Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354145101Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354254504Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354328236Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354362767Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354507911Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354563132Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354586413Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354633064Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354799798Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354858259Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.354992403Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355103115Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355212838Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355323211Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355440784Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355631929Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355798463Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355857945Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355917126Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.355991858Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356049479Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356117731Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356182453Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356284315Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356342746Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35645914Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356564752Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356627694Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356687465Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356778257Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356800708Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35688319Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35689228Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.356935972Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.357126806Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.357348612Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.357516926Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.357802583Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358021459Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3580707Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358142562Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358155982Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358197883Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358273065Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358315026Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358382908Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35843991Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358554872Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358615674Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358699616Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358769758Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35886554Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.358927312Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359085946Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359194248Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359298581Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359359193Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359418984Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359492536Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359548767Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.35966111Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359770243Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359830254Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.359941737Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360017419Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360025539Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360079261Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360198714Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360316637Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360468301Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360593733Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360649335Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360801289Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.360910072Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.361290641Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.361399374Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36164785Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.361872826Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362016829Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362272376Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362330057Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362405239Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36245725Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362524822Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362580703Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362643045Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362703997Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36283264Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.362941723Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363009144Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363063506Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363128207Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363183659Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363284841Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363294361Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363368583Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363563578Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.363874286Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36403145Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.364236125Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36444561Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.364455571Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.364504872Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/aws-ebs-csi-driver-node-4cjsk.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.364611505Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-driver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.364657596Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-driver/csi-driver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.364665416Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-driver/csi-driver/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.364724177Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365118967Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36521885Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365263301Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-liveness-probe/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365270871Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365279081Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365350543Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365461226Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365551468Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36561279Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-node-driver-registrar/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36561998Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36563235Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365692842Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365848806Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365934968Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-4cjsk/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.365970639Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36601994Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/aws-ebs-csi-driver-node-hkk4c.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366111322Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-driver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366130863Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-driver/csi-driver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366138183Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-driver/csi-driver/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366194144Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366310077Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36641043Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36642447Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-liveness-probe/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366447901Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366452881Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366525533Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366664476Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366777469Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36681035Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-node-driver-registrar/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36681705Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36682323Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.366919803Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367015985Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367086347Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hkk4c/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367140118Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367188239Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/aws-ebs-csi-driver-node-zkwhf.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367281032Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-driver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367292832Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-driver/csi-driver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367302282Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-driver/csi-driver/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367346683Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367439216Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367522928Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367575829Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-liveness-probe/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367582939Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367587069Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367633101Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367765984Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367861366Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367919728Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-node-driver-registrar/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367927928Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.367937508Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36799255Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368124333Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368222885Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zkwhf/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368254306Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368310708Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368370109Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368500723Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368561094Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368611285Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368660196Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368724338Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.36878949Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368859761Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368868282Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368872492Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368917143Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368986055Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.368993915Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.369036156Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.369108668Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.369122098Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.369170389Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.369328973Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.369509268Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.369649111Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.369905028Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370106973Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370154994Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370223626Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370269957Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370362419Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370418921Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370555684Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370632826Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370727178Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37081349Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370862932Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370947134Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.370975975Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371083227Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371110688Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371218171Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371251852Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371290533Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371380545Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371414536Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371423196Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371481567Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37157242Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.371614041Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.373075947Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.373223681Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.373381435Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.373476248Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.373691293Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.373912939Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374060872Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374129224Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374166135Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374222706Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374260327Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374331069Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37436307Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374433522Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374471893Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374558555Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.374638697Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375041787Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375091418Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3751656Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375199341Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375280663Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375308294Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375377335Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375416396Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375465438Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375475288Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375536449Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.375763725Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37596868Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376115164Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37635729Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376582756Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376590886Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-f5tlf/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376631917Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-f5tlf/tuned-f5tlf.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376797881Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-f5tlf/tuned/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376813532Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-f5tlf/tuned/tuned/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376818492Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-f5tlf/tuned/tuned/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376827242Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-f5tlf/tuned/tuned/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.376961535Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-f5tlf/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377040757Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-f5tlf/tuned/tuned/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377103949Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-t5xp7/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377120599Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-t5xp7/tuned-t5xp7.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377226792Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-t5xp7/tuned/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377237862Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-t5xp7/tuned/tuned/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377258273Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-t5xp7/tuned/tuned/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377270523Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-t5xp7/tuned/tuned/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377381456Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-t5xp7/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377453877Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-t5xp7/tuned/tuned/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377467008Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tncc8/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377524919Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tncc8/tuned-tncc8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377613792Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tncc8/tuned/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377623062Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tncc8/tuned/tuned/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377627852Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tncc8/tuned/tuned/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377638672Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tncc8/tuned/tuned/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377787766Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tncc8/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377877598Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tncc8/tuned/tuned/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377894899Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3779457Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.377993711Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378039042Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378085984Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378101724Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378160175Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-133-215.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378248377Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-136-111.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37833346Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-137-238.ec2.internal.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378381421Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378424552Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378516434Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378550345Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378640277Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378676698Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37875697Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378809302Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.378892704Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379017027Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37912441Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379181501Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379217412Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379275523Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379294644Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379419577Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379461918Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379507979Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379590071Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379673343Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.379746955Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.37994805Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380035993Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380174756Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380254348Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380379401Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380497684Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380678629Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380760281Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380810562Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380889944Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380906654Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.380998927Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381023127Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.38112694Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381198612Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381239623Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381282024Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381333255Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381375706Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381476909Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.38152162Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381574011Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381616752Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381666334Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381673254Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381706744Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.381977161Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.382210177Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.382382841Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.382585817Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.382850363Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.382865224Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.382898234Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator-6748489656-ssvr9.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383005147Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator-watch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383013757Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383017988Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383064359Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383158441Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383231583Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383290604Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383299595Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383311255Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.383362896Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384427083Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384490654Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ssvr9/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384541866Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384573477Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384642668Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384686759Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384807562Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384860734Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384923376Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.384952106Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385026448Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385068719Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385164891Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385255074Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385390167Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385449169Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.38550861Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385574812Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385611073Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385731536Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385814888Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385864619Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385937951Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.385993572Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.386001632Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.386045574Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.386134106Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.386214578Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.386259299Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.386382562Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.386505535Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.387349826Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.387370567Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.387377147Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.38747885Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.387651094Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.387732126Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.387808728Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.38788071Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.387930591Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.387999243Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388039604Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388107165Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388131456Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388228139Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388306381Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388365312Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388415023Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388478545Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388524456Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388617528Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.38867469Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388730061Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388780322Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388869015Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388879225Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.388931826Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.38908713Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.389266054Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.389648584Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390019044Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390210008Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390218118Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-ddr24/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390253269Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-ddr24/volume-data-source-validator-549c6d54cf-ddr24.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390335081Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-ddr24/volume-data-source-validator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390342682Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-ddr24/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390347132Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-ddr24/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390380542Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-ddr24/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390485415Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-ddr24/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390549857Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-ddr24/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390592728Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390629569Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39068065Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39068769Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390743322Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390818633Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390866135Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390936587Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.390964857Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39106285Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39108968Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391174922Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391222334Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391259345Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391303286Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391386988Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3914637Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391543242Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391599383Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391647594Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391700866Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391783108Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39186105Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391922772Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.391956122Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392045304Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392100316Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392107246Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392159077Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392235339Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39228311Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392386043Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392467365Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392572508Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39264781Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392752652Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.392892396Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39306703Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393137502Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393188313Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393251885Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393309086Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393344337Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393381588Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39346143Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393548552Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393581943Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393631804Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393688736Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393767238Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3938316Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39384106Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.393900731Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.394066105Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.394232919Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.394370683Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.394570218Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.394803994Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.394843795Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.394906956Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.394938937Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39504872Z namespaces/openshift-config-managed/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395081371Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395257535Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395297466Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395382548Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395406559Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395497041Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395574103Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395653035Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395699976Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395769808Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39583517Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.39585193Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.395955653Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.396011714Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.396050695Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.396129657Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.396209179Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.3962558Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.399594674Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.399676596Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400077217Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400158908Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400243391Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400394585Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400504047Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400570169Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40061833Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400738963Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400806265Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400840786Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400894257Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.400936248Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40100727Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401047911Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401139113Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401217575Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401263496Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401309048Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401365599Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40141147Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401466501Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401475522Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401528043Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401739608Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.401965874Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.402117938Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.402302582Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40259745Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40261681Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.402678482Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.402689992Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.402777864Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.402847396Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.402890027Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.402966379Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40299956Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403061512Z namespaces/openshift-config-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403071602Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403079812Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403123523Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403207695Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403224796Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403282417Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403431331Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403608025Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40379075Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.403994865Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.404504258Z namespaces/openshift-config/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.404552369Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.404620701Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.404652941Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.404738904Z namespaces/openshift-config/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.404815616Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.404888927Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4049729Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405055502Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405116273Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405173815Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405249536Z namespaces/openshift-config/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405297688Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4053806Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405425991Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405458622Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405544634Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405591355Z namespaces/openshift-config/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405635276Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4058015Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405887932Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.405967965Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406047287Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406127538Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406274782Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406429956Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406466757Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406509208Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406558029Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406602221Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.406959419Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40699209Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407054452Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407088503Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407182165Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407259597Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407289868Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407343899Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407398981Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407446192Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407501173Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407530204Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407567105Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407732449Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.407936774Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.408077138Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.408270103Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.408454547Z namespaces/openshift-config/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.408501278Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40855226Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40857858Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.408681203Z namespaces/openshift-console-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.408734954Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.408817026Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.408851067Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40894997Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.40897836Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409076103Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409203796Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409310889Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409339799Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409391801Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409438562Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409464753Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409556115Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409602686Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409629267Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409724229Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409800751Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409808261Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409855652Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409918484Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.409962585Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41055425Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.410640012Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411034072Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411112644Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411237107Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41136209Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411543715Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411602506Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411639957Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411698189Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41174661Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411831322Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411847193Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.411938055Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412024467Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412089579Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412108459Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412180611Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412221122Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412286344Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412306114Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412369766Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412412177Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412456028Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412465418Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412525839Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412694004Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.412908469Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.413046993Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.413243308Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.413527175Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.413536035Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-2gspt/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.413581076Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-2gspt/console-operator-584dcb6b54-2gspt.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.413680679Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-2gspt/console-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41374098Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-2gspt/console-operator/console-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41374934Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-2gspt/console-operator/console-operator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.413771261Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-2gspt/console-operator/console-operator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.414293554Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-2gspt/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.414311144Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.414371386Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.414446318Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.414461438Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41453361Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41454467Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4145507Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.414592411Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.414878799Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.415082344Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.415220657Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.415436193Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.415651528Z namespaces/openshift-console/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.415688859Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416076319Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41613586Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416194452Z namespaces/openshift-console/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416241383Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416322225Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416492009Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416807247Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416838728Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416895609Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.416961751Z namespaces/openshift-console/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.417008592Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.417089754Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.417136506Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.417192047Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.417276809Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.417332151Z namespaces/openshift-console/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.417361071Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.418025718Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41811735Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.418393587Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.418468349Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.418659214Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.418812698Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.418994582Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.419060474Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.419118435Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.419203778Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.419264359Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.419632698Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.41970095Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.419787932Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.419891875Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.419936946Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420005728Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420066579Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420157831Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420181352Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420264954Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420303015Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420316755Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420369157Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420540581Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420773747Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.420926411Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.421147446Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.421370472Z namespaces/openshift-console/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.421390073Z namespaces/openshift-console/pods/console-657d64dcd5-52p67/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.421460134Z namespaces/openshift-console/pods/console-657d64dcd5-52p67/console-657d64dcd5-52p67.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.421951947Z namespaces/openshift-console/pods/console-657d64dcd5-52p67/console/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422003788Z namespaces/openshift-console/pods/console-657d64dcd5-52p67/console/console/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422010538Z namespaces/openshift-console/pods/console-657d64dcd5-52p67/console/console/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422016318Z namespaces/openshift-console/pods/console-657d64dcd5-52p67/console/console/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422022188Z namespaces/openshift-console/pods/console-657d64dcd5-52p67/console/console/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422028149Z namespaces/openshift-console/pods/console-657d64dcd5-52p67/console/console/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422033778Z namespaces/openshift-console/pods/downloads-7c97dc4555-mr525/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422044369Z namespaces/openshift-console/pods/downloads-7c97dc4555-mr525/downloads-7c97dc4555-mr525.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422067999Z namespaces/openshift-console/pods/downloads-7c97dc4555-mr525/download-server/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42207856Z namespaces/openshift-console/pods/downloads-7c97dc4555-mr525/download-server/download-server/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42208446Z namespaces/openshift-console/pods/downloads-7c97dc4555-mr525/download-server/download-server/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422143211Z namespaces/openshift-console/pods/downloads-7c97dc4555-mr525/download-server/download-server/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422601153Z namespaces/openshift-console/pods/downloads-7c97dc4555-mr525/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422668454Z namespaces/openshift-console/pods/downloads-7c97dc4555-mr525/download-server/download-server/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422737706Z namespaces/openshift-console/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422752747Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422832709Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.422850859Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423005013Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423060974Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423142437Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423180007Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423233049Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42326768Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423358482Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423434464Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423515926Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423564767Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423605888Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423653379Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423736951Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423840534Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423904786Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.423938317Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424014248Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424098221Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424145112Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424259535Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424337476Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424423619Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424497961Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424576763Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424728917Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424921821Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.424943512Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425011684Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425060585Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425086675Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425176738Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425201488Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42528848Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425382123Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425407243Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425453985Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425473545Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425533317Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425578148Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425586688Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425617839Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.425823924Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.426248545Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.426387058Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.426585623Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.426808499Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.426825199Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.426887671Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.426935202Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427019914Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427053745Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427099666Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427154898Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427204479Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427218879Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427323352Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427398753Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427479926Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427564278Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427619939Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427682851Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427753893Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427842135Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427877425Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.427926397Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428001389Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42804914Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428090511Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428142872Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428229805Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428271875Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428378248Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42845039Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428532572Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428614104Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.428688856Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42886163Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429010014Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429152678Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429195889Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42924474Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429286701Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429335532Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429375103Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429459205Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429531677Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429571438Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429619309Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.42966391Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429726972Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429797934Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429807394Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.429846355Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.430006179Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.430178814Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.430311677Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.430514062Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.430721267Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.430758468Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43082781Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43084242Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.430951793Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.430995684Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431184579Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4312233Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431296411Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431325342Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431407084Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431482436Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431558338Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431616959Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431659851Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431704512Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431851336Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431933518Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.431993819Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43203032Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432113792Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432192604Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432236915Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432330688Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43240791Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432497602Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432567503Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432649405Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432787459Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.432976814Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433080816Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433120728Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433183419Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43320973Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433258571Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433305412Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433389254Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43360729Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433649191Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433679311Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433762724Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433825125Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433876096Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433914597Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.433970259Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43400048Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.434044061Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.434053781Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.434090542Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.434245186Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43443665Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.434572594Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43479505Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.434996635Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435032745Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435098227Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435138218Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435229021Z namespaces/openshift-dns/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435255961Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435332863Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435371714Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435930658Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.435964109Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436126213Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436226595Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436329078Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436363329Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43639683Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436474612Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436524603Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436603035Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436659917Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436696017Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43681161Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436881142Z namespaces/openshift-dns/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.436924443Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.437037346Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.437121188Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.437351204Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.437429406Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.437698532Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.437855356Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438034581Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438101813Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438149774Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438212616Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438260067Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438328158Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438376489Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438458192Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438538324Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438610366Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438656137Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438703458Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.43877715Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438822651Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438867532Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438923563Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438935264Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.438966154Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.439123568Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.439469617Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.439609591Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.439835706Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440094443Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440104653Z namespaces/openshift-dns/pods/dns-default-5g4wm/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440156284Z namespaces/openshift-dns/pods/dns-default-5g4wm/dns-default-5g4wm.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440246767Z namespaces/openshift-dns/pods/dns-default-5g4wm/dns/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440256987Z namespaces/openshift-dns/pods/dns-default-5g4wm/dns/dns/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440266587Z namespaces/openshift-dns/pods/dns-default-5g4wm/dns/dns/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440307108Z namespaces/openshift-dns/pods/dns-default-5g4wm/dns/dns/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440402191Z namespaces/openshift-dns/pods/dns-default-5g4wm/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440473222Z namespaces/openshift-dns/pods/dns-default-5g4wm/dns/dns/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440506993Z namespaces/openshift-dns/pods/dns-default-5g4wm/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440512113Z namespaces/openshift-dns/pods/dns-default-5g4wm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440518323Z namespaces/openshift-dns/pods/dns-default-5g4wm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440574635Z namespaces/openshift-dns/pods/dns-default-5g4wm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440666237Z namespaces/openshift-dns/pods/dns-default-5g4wm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440823041Z namespaces/openshift-dns/pods/dns-default-5g4wm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440841482Z namespaces/openshift-dns/pods/dns-default-clksn/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440898383Z namespaces/openshift-dns/pods/dns-default-clksn/dns-default-clksn.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440982025Z namespaces/openshift-dns/pods/dns-default-clksn/dns/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440990425Z namespaces/openshift-dns/pods/dns-default-clksn/dns/dns/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.440998126Z namespaces/openshift-dns/pods/dns-default-clksn/dns/dns/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441044267Z namespaces/openshift-dns/pods/dns-default-clksn/dns/dns/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441135539Z namespaces/openshift-dns/pods/dns-default-clksn/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441204111Z namespaces/openshift-dns/pods/dns-default-clksn/dns/dns/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441252092Z namespaces/openshift-dns/pods/dns-default-clksn/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441258982Z namespaces/openshift-dns/pods/dns-default-clksn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441271473Z namespaces/openshift-dns/pods/dns-default-clksn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441311554Z namespaces/openshift-dns/pods/dns-default-clksn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441404216Z namespaces/openshift-dns/pods/dns-default-clksn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441471418Z namespaces/openshift-dns/pods/dns-default-clksn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441529719Z namespaces/openshift-dns/pods/dns-default-wfpdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44156249Z namespaces/openshift-dns/pods/dns-default-wfpdb/dns-default-wfpdb.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441648012Z namespaces/openshift-dns/pods/dns-default-wfpdb/dns/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441657782Z namespaces/openshift-dns/pods/dns-default-wfpdb/dns/dns/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441666222Z namespaces/openshift-dns/pods/dns-default-wfpdb/dns/dns/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441723134Z namespaces/openshift-dns/pods/dns-default-wfpdb/dns/dns/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441830487Z namespaces/openshift-dns/pods/dns-default-wfpdb/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441910578Z namespaces/openshift-dns/pods/dns-default-wfpdb/dns/dns/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.441944609Z namespaces/openshift-dns/pods/dns-default-wfpdb/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4419543Z namespaces/openshift-dns/pods/dns-default-wfpdb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44196378Z namespaces/openshift-dns/pods/dns-default-wfpdb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442020401Z namespaces/openshift-dns/pods/dns-default-wfpdb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442105363Z namespaces/openshift-dns/pods/dns-default-wfpdb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442175285Z namespaces/openshift-dns/pods/dns-default-wfpdb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442206706Z namespaces/openshift-dns/pods/node-resolver-7tk7q/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442234667Z namespaces/openshift-dns/pods/node-resolver-7tk7q/node-resolver-7tk7q.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442334229Z namespaces/openshift-dns/pods/node-resolver-7tk7q/dns-node-resolver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442345129Z namespaces/openshift-dns/pods/node-resolver-7tk7q/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44235488Z namespaces/openshift-dns/pods/node-resolver-7tk7q/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442390081Z namespaces/openshift-dns/pods/node-resolver-7tk7q/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442478853Z namespaces/openshift-dns/pods/node-resolver-7tk7q/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442544305Z namespaces/openshift-dns/pods/node-resolver-7tk7q/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442590716Z namespaces/openshift-dns/pods/node-resolver-92rx8/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442639257Z namespaces/openshift-dns/pods/node-resolver-92rx8/node-resolver-92rx8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44274353Z namespaces/openshift-dns/pods/node-resolver-92rx8/dns-node-resolver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44278354Z namespaces/openshift-dns/pods/node-resolver-92rx8/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44278764Z namespaces/openshift-dns/pods/node-resolver-92rx8/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442798991Z namespaces/openshift-dns/pods/node-resolver-92rx8/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442881263Z namespaces/openshift-dns/pods/node-resolver-92rx8/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442949775Z namespaces/openshift-dns/pods/node-resolver-92rx8/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.442990636Z namespaces/openshift-dns/pods/node-resolver-jh65p/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443019977Z namespaces/openshift-dns/pods/node-resolver-jh65p/node-resolver-jh65p.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443099959Z namespaces/openshift-dns/pods/node-resolver-jh65p/dns-node-resolver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443110159Z namespaces/openshift-dns/pods/node-resolver-jh65p/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443116099Z namespaces/openshift-dns/pods/node-resolver-jh65p/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443125189Z namespaces/openshift-dns/pods/node-resolver-jh65p/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443215951Z namespaces/openshift-dns/pods/node-resolver-jh65p/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443291253Z namespaces/openshift-dns/pods/node-resolver-jh65p/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443311334Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443362745Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443410186Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443436347Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44353803Z namespaces/openshift-etcd/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443584961Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443639832Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443671283Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443761925Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443796456Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.443886388Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44396453Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444044912Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444079033Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444128474Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444174106Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444204366Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444294829Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44434675Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444383921Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444473803Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444547065Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444592576Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444696568Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444800641Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444880963Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.444962515Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445039457Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4451636Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445323284Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445381876Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445433167Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445481718Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445525989Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445570021Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445612462Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445694874Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445916859Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.445976021Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446025042Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446070693Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446092494Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446150345Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446164756Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446208406Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44636187Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446550235Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446685849Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.446908814Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447104649Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44714526Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447199851Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447246113Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447468518Z namespaces/openshift-host-network/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447517399Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447573611Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447613802Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447694454Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447750745Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.447845268Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44792538Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448007152Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448068073Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448120554Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448175006Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448218667Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448299999Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44835041Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448403932Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448476294Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448579216Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448593006Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44872696Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448814122Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448892074Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.448971736Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449052688Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449194912Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449297184Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449346595Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449390706Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449456738Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44950908Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449563261Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449626172Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449705645Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449762856Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449861078Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.44994041Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.449978821Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450031203Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450080174Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450144225Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450197427Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450215237Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450268039Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450429953Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450628558Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.450801912Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451008807Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451205442Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451248273Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451298475Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451341066Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451420938Z namespaces/openshift-image-registry/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451465589Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4515306Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451566511Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451625803Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451658584Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.451795477Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.45192274Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452121875Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452162706Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452203207Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452250158Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452287119Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452393822Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452450834Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452492595Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452578947Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452623338Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452631258Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.452669989Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.453034978Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.453079919Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.454350181Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.454433773Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.45469337Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.454804783Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455018308Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455162412Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455352687Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455412228Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455454099Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455510691Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455550061Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455604983Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455650714Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455786487Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455863259Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455924851Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.455972772Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456035474Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456080375Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456181877Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.45627819Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456335251Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456381582Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456435654Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456484425Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456524506Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456547497Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456596308Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456794903Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.456972437Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457118671Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457328386Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457524961Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457535101Z namespaces/openshift-image-registry/pods/image-registry-776cdb854c-qdpln/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457564722Z namespaces/openshift-image-registry/pods/image-registry-776cdb854c-qdpln/image-registry-776cdb854c-qdpln.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457670545Z namespaces/openshift-image-registry/pods/image-registry-776cdb854c-qdpln/registry/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457678065Z namespaces/openshift-image-registry/pods/image-registry-776cdb854c-qdpln/registry/registry/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457684525Z namespaces/openshift-image-registry/pods/image-registry-776cdb854c-qdpln/registry/registry/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.457771007Z namespaces/openshift-image-registry/pods/image-registry-776cdb854c-qdpln/registry/registry/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460075075Z namespaces/openshift-image-registry/pods/image-registry-776cdb854c-qdpln/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460139517Z namespaces/openshift-image-registry/pods/image-registry-776cdb854c-qdpln/registry/registry/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460175868Z namespaces/openshift-image-registry/pods/node-ca-4w8n8/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460207349Z namespaces/openshift-image-registry/pods/node-ca-4w8n8/node-ca-4w8n8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460294201Z namespaces/openshift-image-registry/pods/node-ca-4w8n8/node-ca/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460304041Z namespaces/openshift-image-registry/pods/node-ca-4w8n8/node-ca/node-ca/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460308901Z namespaces/openshift-image-registry/pods/node-ca-4w8n8/node-ca/node-ca/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460366303Z namespaces/openshift-image-registry/pods/node-ca-4w8n8/node-ca/node-ca/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460552057Z namespaces/openshift-image-registry/pods/node-ca-4w8n8/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460627269Z namespaces/openshift-image-registry/pods/node-ca-4w8n8/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46064471Z namespaces/openshift-image-registry/pods/node-ca-cxrrh/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460703601Z namespaces/openshift-image-registry/pods/node-ca-cxrrh/node-ca-cxrrh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460808464Z namespaces/openshift-image-registry/pods/node-ca-cxrrh/node-ca/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460821254Z namespaces/openshift-image-registry/pods/node-ca-cxrrh/node-ca/node-ca/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460846855Z namespaces/openshift-image-registry/pods/node-ca-cxrrh/node-ca/node-ca/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.460873625Z namespaces/openshift-image-registry/pods/node-ca-cxrrh/node-ca/node-ca/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46106376Z namespaces/openshift-image-registry/pods/node-ca-cxrrh/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461131412Z namespaces/openshift-image-registry/pods/node-ca-cxrrh/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461148372Z namespaces/openshift-image-registry/pods/node-ca-qkksk/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461201274Z namespaces/openshift-image-registry/pods/node-ca-qkksk/node-ca-qkksk.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461275725Z namespaces/openshift-image-registry/pods/node-ca-qkksk/node-ca/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461288056Z namespaces/openshift-image-registry/pods/node-ca-qkksk/node-ca/node-ca/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461299646Z namespaces/openshift-image-registry/pods/node-ca-qkksk/node-ca/node-ca/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461349607Z namespaces/openshift-image-registry/pods/node-ca-qkksk/node-ca/node-ca/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461528582Z namespaces/openshift-image-registry/pods/node-ca-qkksk/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461592184Z namespaces/openshift-image-registry/pods/node-ca-qkksk/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461634885Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461701006Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46187128Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461909952Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461961283Z namespaces/openshift-infra/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461969583Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461977843Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.461999844Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.462158798Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.462339762Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.462476886Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.462858275Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46307095Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463118012Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463187013Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463229705Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463300096Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463330737Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4634448Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463524352Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463606894Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463659115Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463734167Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46385677Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463905082Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.463997124Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464038055Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464083016Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464160728Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46424539Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464291911Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464403374Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464486666Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464665721Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464770463Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.464942428Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465076441Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465239125Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465291746Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465337367Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465378879Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46542969Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465497082Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465541973Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465629065Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465720917Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465780809Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465837Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465881461Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465923782Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465980584Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.465997474Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.466044045Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.466193129Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.466374194Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.466509987Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.466741573Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467000029Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46700957Z namespaces/openshift-ingress-canary/pods/ingress-canary-48qh4/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467054431Z namespaces/openshift-ingress-canary/pods/ingress-canary-48qh4/ingress-canary-48qh4.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467138403Z namespaces/openshift-ingress-canary/pods/ingress-canary-48qh4/serve-healthcheck-canary/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467152473Z namespaces/openshift-ingress-canary/pods/ingress-canary-48qh4/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467156613Z namespaces/openshift-ingress-canary/pods/ingress-canary-48qh4/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467203655Z namespaces/openshift-ingress-canary/pods/ingress-canary-48qh4/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467308297Z namespaces/openshift-ingress-canary/pods/ingress-canary-48qh4/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46741702Z namespaces/openshift-ingress-canary/pods/ingress-canary-48qh4/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467522703Z namespaces/openshift-ingress-canary/pods/ingress-canary-dxmmk/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467607045Z namespaces/openshift-ingress-canary/pods/ingress-canary-dxmmk/ingress-canary-dxmmk.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467759688Z namespaces/openshift-ingress-canary/pods/ingress-canary-dxmmk/serve-healthcheck-canary/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46781226Z namespaces/openshift-ingress-canary/pods/ingress-canary-dxmmk/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46782457Z namespaces/openshift-ingress-canary/pods/ingress-canary-dxmmk/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467844311Z namespaces/openshift-ingress-canary/pods/ingress-canary-dxmmk/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.467986124Z namespaces/openshift-ingress-canary/pods/ingress-canary-dxmmk/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468083687Z namespaces/openshift-ingress-canary/pods/ingress-canary-dxmmk/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468367234Z namespaces/openshift-ingress-canary/pods/ingress-canary-rrkx5/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468418155Z namespaces/openshift-ingress-canary/pods/ingress-canary-rrkx5/ingress-canary-rrkx5.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468520578Z namespaces/openshift-ingress-canary/pods/ingress-canary-rrkx5/serve-healthcheck-canary/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468531068Z namespaces/openshift-ingress-canary/pods/ingress-canary-rrkx5/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468578209Z namespaces/openshift-ingress-canary/pods/ingress-canary-rrkx5/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46861399Z namespaces/openshift-ingress-canary/pods/ingress-canary-rrkx5/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468769454Z namespaces/openshift-ingress-canary/pods/ingress-canary-rrkx5/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468867106Z namespaces/openshift-ingress-canary/pods/ingress-canary-rrkx5/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468917548Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.468971099Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.469055601Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.469107962Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.469258876Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.469317578Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.46941861Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.469472742Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.469560224Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.469622385Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.469733028Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.470008155Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.470121438Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47020817Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.470268572Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.470359624Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.470417765Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.470548589Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47059965Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.470668502Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.470804465Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.471104893Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.471120513Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.472574809Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.472663152Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.472842626Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.472921588Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4729932Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473126923Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473359609Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473431751Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473497833Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473543994Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473600285Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473656817Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473668137Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473745909Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473847441Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473886613Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.473992455Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474052187Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474115618Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474160149Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474215471Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474256682Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474332554Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474377315Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474442216Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474472647Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474535529Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474547859Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47459729Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474689293Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474728294Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.474805056Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.475036871Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.475289938Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.475457542Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.475704218Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.476020126Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.476081408Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47615719Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.476239042Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.476372295Z namespaces/openshift-ingress/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.476432117Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.476833457Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.476897398Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47697882Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.477025081Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.477118484Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47736744Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.477569835Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.477622736Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.477672218Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47776212Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.477826012Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.477906604Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.477962985Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.478009426Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.478085778Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47814636Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.47815948Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.478218211Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.478304384Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.478381916Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.478496918Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.478527759Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.479008141Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.479148985Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.479451893Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.479563415Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.479914674Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480083238Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480351305Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480480108Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.4805346Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480641143Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480652293Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480731775Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480833617Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480889209Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.480987251Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481045862Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481148515Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481260548Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481321029Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481383431Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481466413Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481516734Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481602076Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481648518Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.48173419Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481765271Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481837202Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481849393Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.481901314Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.482052388Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.482230972Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.482370936Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.482970581Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.483234768Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.483249868Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.483289359Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/istiod-openshift-gateway-5677ccb49b-x4v6g.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.483452093Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.483466214Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.483485484Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.483536685Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.496777018Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.49685477Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-x4v6g/discovery/discovery/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.496931152Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.496981763Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.497453915Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.497468355Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.497475396Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.497519477Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500190214Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-nvf9s/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500282416Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500328227Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router-default-5dc8cf7cff-gn2tw.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500476061Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500486822Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500494092Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500535413Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500658256Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500743278Z namespaces/openshift-ingress/pods/router-default-5dc8cf7cff-gn2tw/router/router/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500794469Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.50083567Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500890542Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.500945063Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501043496Z namespaces/openshift-insights/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501099527Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501174659Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.50121647Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501289292Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501326903Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501445216Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501555149Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501655281Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501701862Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501764264Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501885967Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.501910267Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.50202089Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.502059851Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.502150793Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.502208575Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.502252106Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.502349878Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.50240964Z namespaces/openshift-insights/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.502457971Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.503406855Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.503504318Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.503819825Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.503956779Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.504495412Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.504582545Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.504783259Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.504843751Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.504893122Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.504985755Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505046166Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505165349Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505240641Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505294182Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505377425Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505432446Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505513708Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505570639Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505651221Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505735763Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.505875157Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.50598366Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506053271Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506109513Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506201155Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506259237Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506350259Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5064117Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506498383Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506556034Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506615746Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506646646Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506722248Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.506984775Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.507288042Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.507464417Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.507765434Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.508044421Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.508055092Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-zxbld/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.508099343Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-zxbld/insights-operator-5885fdb759-zxbld.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.508225446Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-zxbld/insights-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.508253607Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-zxbld/insights-operator/insights-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.508262847Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-zxbld/insights-operator/insights-operator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.508325139Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-zxbld/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.508857062Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-zxbld/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509219371Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509270823Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/insights-runtime-extractor-6ffmn.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509550999Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.50956084Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/exporter/exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.50958409Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/exporter/exporter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509609631Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/exporter/exporter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509760605Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509891948Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/exporter/exporter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509924959Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/extractor/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509935779Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/extractor/extractor/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.509941749Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/extractor/extractor/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510006101Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/extractor/extractor/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510137974Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510241947Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/extractor/extractor/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510275468Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510287788Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510307129Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510390171Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510526364Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510630757Z namespaces/openshift-insights/pods/insights-runtime-extractor-6ffmn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510706819Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510878953Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/insights-runtime-extractor-8dpn4.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.510986075Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511020096Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/exporter/exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511032907Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/exporter/exporter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511126839Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/exporter/exporter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511283513Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511394556Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/exporter/exporter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511427597Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/extractor/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511498508Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/extractor/extractor/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511516309Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/extractor/extractor/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.51156557Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/extractor/extractor/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511725364Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511836277Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/extractor/extractor/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.511884258Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512102704Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512110344Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512171895Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512303399Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512408261Z namespaces/openshift-insights/pods/insights-runtime-extractor-8dpn4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512465083Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512528044Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/insights-runtime-extractor-fg6ls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512659128Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512675448Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/exporter/exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512681718Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/exporter/exporter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512791331Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/exporter/exporter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512864823Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.512960735Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/exporter/exporter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513012486Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/extractor/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513034407Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/extractor/extractor/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513041297Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/extractor/extractor/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513099359Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/extractor/extractor/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513228272Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513323544Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/extractor/extractor/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513370215Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513382126Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513390216Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513452268Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513567651Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513661853Z namespaces/openshift-insights/pods/insights-runtime-extractor-fg6ls/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513723934Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513811617Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.513876098Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.51393237Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514003731Z namespaces/openshift-keda/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514012942Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514025522Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514089513Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514214967Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514271308Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514280018Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.51433609Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514409472Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514470713Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514591416Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514701889Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514800471Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514816792Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.514872133Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.515059588Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.515280823Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.515573101Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.515876418Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516063873Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516271088Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.51632686Z namespaces/openshift-keda/operators.coreos.com/installplans/install-w5l9z.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516483974Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516522855Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516607017Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516641428Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516758531Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516797892Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516917385Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.516974016Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517056248Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517110129Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517184191Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517238183Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517351406Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517459138Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517567011Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517629863Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517690564Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517798987Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517858388Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.517959701Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518017492Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518081594Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518201317Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518276479Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.51833542Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518468804Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518558736Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518649898Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518800152Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.518886474Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519030578Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519248593Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519315045Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519374886Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519434268Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519482079Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519551131Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519582042Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519703275Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519879379Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.51992732Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.519984412Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.520039913Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.520099015Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.520170937Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.520181407Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.520245508Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.520434833Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.52109368Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.521265024Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.521528141Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.521787087Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.521846939Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.52191585Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.521968942Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522064684Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522128616Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522221578Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522260739Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522338781Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522397022Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522510145Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522620518Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522752001Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522816903Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522888905Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522937136Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.522983007Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523060459Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523111411Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523158132Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523239264Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523286255Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523331336Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523441749Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523514401Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523582882Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523677425Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523830089Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.523951252Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524097785Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524149157Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524187798Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524238869Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.52428446Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524340391Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524369672Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524460954Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524528836Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524583317Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524629609Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.52469637Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524760992Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524874425Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524911536Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.524982698Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.52505996Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.525108001Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.525179772Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.525209223Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.525260074Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.525456819Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.525684765Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.525904961Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526164787Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526387133Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526426654Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526492596Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526544407Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526640509Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526702221Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526836304Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526878315Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.526954367Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527008089Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527118711Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527224904Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527330627Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527389728Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.52744978Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527516151Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527582553Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527701666Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527778488Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.52784295Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.527955072Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.528023594Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.528082136Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.528216659Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.528328502Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.528435344Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.528544147Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.52865402Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.528845355Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529076761Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529160353Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529225064Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529288876Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529344437Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529414669Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529474591Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529585023Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.529690336Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530044415Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530103686Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530181718Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.53023983Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530310181Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530376113Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530433945Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530443895Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530514457Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530700871Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.530950768Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.531132442Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.531392929Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.531611504Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.531671906Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.531882381Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.531934642Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532047835Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532107637Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532181508Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.53224776Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532317692Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532394864Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532484336Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532595969Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532703292Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532803694Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532855596Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532945828Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.532995359Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533110522Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533170603Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533230015Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533348408Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.53341307Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.53342437Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533469441Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533561763Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533613904Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533768058Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.533902592Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.534487766Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.534573119Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.534659501Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.534839315Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535000639Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535044221Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535092672Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535134973Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535178574Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535234425Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535287797Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535379719Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535459131Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535503972Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535546143Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535591554Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535639975Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535679887Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535688747Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535747798Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.535910842Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.536087217Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.536433586Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.53662906Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.536838406Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.536891267Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.536947719Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.536992949Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537087612Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537141513Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537234996Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537281207Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537355629Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5374114Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537519103Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537629245Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537759999Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537849531Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537908963Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.537972834Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538023215Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538135098Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.53819746Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538255461Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538361074Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538439566Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538497177Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538637191Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538771114Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.538881457Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.53898509Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.539098362Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.539259817Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.539486702Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.539554364Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.539617815Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.539682827Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5398068Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.539881772Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.539941984Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540049766Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540154029Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540217361Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540280942Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540336654Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540402305Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540458067Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540467587Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540536189Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.540787285Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.54099527Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.541167614Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.541436111Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.541649366Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.541704718Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.541831861Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.541877452Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.541968075Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542026026Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542096218Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542151269Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542220811Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542283193Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542383185Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542495628Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542600231Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542659002Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542742854Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542815256Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.542877128Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.54298956Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543049192Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543121924Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543216196Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543280578Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543349389Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543485283Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543587686Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543693678Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543825421Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.543941124Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544092638Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544308644Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544376765Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544436117Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544488078Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544540499Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544608921Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544668953Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544781425Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544858767Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544902279Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544944889Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.544989531Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.545033512Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.545076893Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.545085113Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.545143445Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.545448762Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.545625667Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.545802561Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.545992766Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546188301Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546221982Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546272343Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546315534Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546381066Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546424117Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546483348Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546520909Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546580071Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546629932Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546698624Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.546871548Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547010441Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547073703Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547136255Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547200526Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547255138Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.54736991Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547436792Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547496304Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547613027Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547668958Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547684218Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547844873Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547938885Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.547997276Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.54814776Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.548264983Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.548480008Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.548587501Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.548812017Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.54896361Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549194926Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549272868Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.54933537Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549413512Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549463843Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549537225Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549595086Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549761941Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549899244Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.549964386Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.550019087Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.550094679Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5501505Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.550219352Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.550230442Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.550294414Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.550488719Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.550728555Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55092001Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.551171426Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.551400862Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.551408982Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.551474474Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.551883354Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n/kube-storage-version-migrator-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.551899695Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.551929735Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.551970766Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5521322Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552266184Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552313115Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552384677Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552427858Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552487309Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55254024Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552601422Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552620783Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552688784Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552764506Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552841978Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.552950041Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553041643Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553100075Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553119715Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553172496Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553228358Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55331131Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553363001Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553392852Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553466564Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553534836Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553553856Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553672629Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553786572Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553912335Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.553989107Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55410826Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554228853Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554383057Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554420618Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55448634Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554533701Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554567111Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554652304Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554706595Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554826968Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554952511Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.554999922Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.555052624Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.555122465Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.555178137Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.555249129Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.555265029Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55532509Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.555507595Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.555842663Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.556028988Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.556280145Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55650244Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55651064Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.556578382Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/migrator-6865c9967-9t7rs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.556677535Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/graceful-termination/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.556688545Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/graceful-termination/graceful-termination/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.556697955Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.556789287Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55690958Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557010753Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557052714Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/migrator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557063224Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/migrator/migrator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557069094Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/migrator/migrator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557117756Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/migrator/migrator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557245699Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557349601Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-9t7rs/migrator/migrator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557377682Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557442904Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557512405Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557555597Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557648769Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55769786Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557828164Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557866674Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.557958627Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.558033809Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.558148272Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.558337556Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55849946Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.558568932Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.558633374Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.558695705Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.558777137Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55889266Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.558953322Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559019694Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559130886Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559190338Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559198328Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559253209Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559373822Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559491735Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559548257Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.55968947Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.559835254Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.560438479Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.560547162Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.560844529Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561027744Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.56127771Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561358022Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561407214Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561495176Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561538507Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561615059Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.56165742Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561801304Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561893796Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.561959648Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562005378Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.56206901Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562103301Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562175053Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562215464Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562279176Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562290226Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562346137Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562502871Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.562684846Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.56286053Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563069755Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.56325023Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563445035Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563488686Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-4cnzl.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563572068Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563621989Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563678371Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563697701Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563809544Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563843645Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563965508Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.563984598Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-7r4r7/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.56405233Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-7r4r7/lws-controller-manager-659b67c655-7r4r7.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564167133Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-7r4r7/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564181073Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-7r4r7/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564191293Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-7r4r7/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564322187Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-7r4r7/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.56445375Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-7r4r7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564558643Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-7r4r7/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564623574Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-hmff7/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564690376Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-hmff7/lws-controller-manager-659b67c655-hmff7.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564915962Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-hmff7/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564932822Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-hmff7/manager/manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564958193Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-hmff7/manager/manager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.564981343Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-hmff7/manager/manager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.565419244Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-hmff7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.565514727Z namespaces/openshift-lws-operator/pods/lws-controller-manager-659b67c655-hmff7/manager/manager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.565587439Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-4tnc8/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.56564234Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-4tnc8/openshift-lws-operator-fd8ccff4c-4tnc8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.565798614Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-4tnc8/openshift-lws-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.565815084Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-4tnc8/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.565830495Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-4tnc8/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.565874216Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-4tnc8/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566218204Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-4tnc8/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566351888Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-4tnc8/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566403709Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566462501Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566534112Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566602344Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566668646Z namespaces/openshift-machine-api/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566678226Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566682266Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566771498Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.566941663Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567122257Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567262551Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567458686Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5676506Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567659641Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567663861Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567689792Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567824455Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567907287Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567918037Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.567967339Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.568120652Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.568338938Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.568466551Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.568668426Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.568906592Z namespaces/openshift-marketplace/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.568920603Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.568924713Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.568959924Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569033925Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569042655Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569075947Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569177109Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569249221Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569334703Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569402515Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569437856Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569589279Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.569818465Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.57003496Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.570234385Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.570473362Z namespaces/openshift-monitoring/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.570522693Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.570593614Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.570631275Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.570700957Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.570765629Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.570923263Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.571393695Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.571745954Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572053041Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572100392Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572146633Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572194325Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572278757Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572353479Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.57240628Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572491352Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572534573Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.572604595Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.575861857Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.57599661Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.576971525Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.577050937Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.577993551Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.578114254Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.578582535Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.57876227Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.578806901Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.578973425Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579015396Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579056157Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579099659Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579187301Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579262443Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579303053Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579349075Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579482658Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579505269Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579607531Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579625042Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579821037Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.579863948Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.57996805Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.580153055Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.580240597Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.580497014Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.580630637Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58075693Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.580865693Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.580966245Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581054997Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58115723Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581182681Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581275733Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581354525Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581439237Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5815429Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581622402Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581805947Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581902309Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.581989611Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582077993Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582164685Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582250518Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582319179Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5823603Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582418592Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582436732Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582463693Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582653478Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.582933515Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58312506Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.583388686Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.583627262Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.583659943Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.583765186Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.583779136Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.583846508Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584025222Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584044863Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584050913Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584065533Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584163636Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584230068Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584261598Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584267148Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584271249Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58430897Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584402292Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584469843Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584501974Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584508044Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584517595Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584555726Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584640638Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58474164Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584759111Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584763521Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584768341Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584815132Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584900444Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.584966506Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585019227Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585029768Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585033948Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585056678Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585147611Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585211212Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585254923Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585263254Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585268054Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585298874Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585397187Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585466629Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585493819Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58550631Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58551349Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585546991Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585634063Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585702245Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585741266Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-xrstz/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585825788Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-xrstz/cluster-monitoring-operator-548b489cb8-xrstz.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58590932Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-xrstz/cluster-monitoring-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58592189Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-xrstz/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58592835Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-xrstz/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.585957301Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-xrstz/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.586898075Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-xrstz/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.586957856Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-xrstz/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.586982597Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587045358Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-state-metrics-56d9f944b4-klk6b.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587140441Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-main/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587155531Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587196142Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587202922Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587320425Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587397817Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587426458Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-self/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587434298Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587438578Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587483719Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587568452Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587643533Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587678244Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-state-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587691644Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587701055Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.587828498Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588027423Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588079334Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-klk6b/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588125255Z namespaces/openshift-monitoring/pods/metrics-server-6788c89f75-58rcg/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588174037Z namespaces/openshift-monitoring/pods/metrics-server-6788c89f75-58rcg/metrics-server-6788c89f75-58rcg.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588279659Z namespaces/openshift-monitoring/pods/metrics-server-6788c89f75-58rcg/metrics-server/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58829375Z namespaces/openshift-monitoring/pods/metrics-server-6788c89f75-58rcg/metrics-server/metrics-server/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58830781Z namespaces/openshift-monitoring/pods/metrics-server-6788c89f75-58rcg/metrics-server/metrics-server/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58831651Z namespaces/openshift-monitoring/pods/metrics-server-6788c89f75-58rcg/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588419413Z namespaces/openshift-monitoring/pods/metrics-server-6788c89f75-58rcg/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588505455Z namespaces/openshift-monitoring/pods/metrics-server-6788c89f75-58rcg/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588526296Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-pq9fh/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588567106Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-pq9fh/monitoring-plugin-68b7fc9574-pq9fh.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588651709Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-pq9fh/monitoring-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588660289Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-pq9fh/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588666659Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-pq9fh/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588739301Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-pq9fh/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588837043Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-pq9fh/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588902955Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-pq9fh/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588933656Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.588979777Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/node-exporter-9q5sz.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.58908601Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/init-textfile/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.589308155Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/init-textfile/init-textfile/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.589350186Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/init-textfile/init-textfile/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.589936371Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590053704Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590065354Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590074774Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590083615Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590090645Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590096975Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590126366Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590139586Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590152596Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/node-exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590159546Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/node-exporter/node-exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590171297Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/node-exporter/node-exporter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590186577Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590194387Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590201338Z namespaces/openshift-monitoring/pods/node-exporter-9q5sz/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590209008Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590215558Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/node-exporter-mfm74.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590227068Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/init-textfile/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590245699Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/init-textfile/init-textfile/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590252379Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/init-textfile/init-textfile/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590258589Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590374672Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59070143Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590762012Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590767692Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590772872Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590777352Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590788642Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.590795182Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591061469Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/node-exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59108071Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/node-exporter/node-exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59109477Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/node-exporter/node-exporter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59110197Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59110784Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59111407Z namespaces/openshift-monitoring/pods/node-exporter-mfm74/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591123381Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591162482Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/node-exporter-vvzl4.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591294785Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/init-textfile/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591313526Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/init-textfile/init-textfile/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591322726Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/init-textfile/init-textfile/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591349007Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591435189Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59150191Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591545441Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591553901Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591565512Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591630523Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591756157Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591834489Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591852979Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/node-exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591860339Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/node-exporter/node-exporter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591866869Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/node-exporter/node-exporter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.591920151Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592031033Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592096495Z namespaces/openshift-monitoring/pods/node-exporter-vvzl4/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592147256Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592182977Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/openshift-state-metrics-b8f49f5c6-mzwvz.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592324971Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-main/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592333171Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592337821Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592380472Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592476285Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592544577Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592582258Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-self/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592589938Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592601388Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592644009Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592789053Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592858254Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592897015Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/openshift-state-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592907316Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592913716Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.592976687Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593098371Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593175282Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-mzwvz/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593245564Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593329306Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593543102Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593551622Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593566802Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593630204Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593787308Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59386152Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593908731Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593916141Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593923811Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.593990853Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594109496Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594195788Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594224909Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594231929Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594235809Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59427818Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594375113Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594443654Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594486415Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594496726Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594501226Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594508366Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594599928Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59467975Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594694961Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594704011Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594733532Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594797863Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594913766Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.594992878Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.595033479Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.595039049Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.595042819Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.595088861Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596159497Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596235669Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596297041Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596307121Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596311531Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596355412Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596475195Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596545377Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596589328Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59664422Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/prometheus-operator-84d69f794d-mhmsz.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596796064Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596812114Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596818704Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596846905Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.596965498Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59704321Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597084991Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/prometheus-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597092751Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597104511Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597148092Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597430309Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597503281Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-mhmsz/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597548022Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-knbzg/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597615834Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-knbzg/prometheus-operator-admission-webhook-7f8d78f54b-knbzg.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597788989Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-knbzg/prometheus-operator-admission-webhook/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597803419Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-knbzg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597809569Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-knbzg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.5978551Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-knbzg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.597954613Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-knbzg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598023184Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-knbzg/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598065765Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598101136Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/telemeter-client-f884997dd-7crk9.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59825849Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59826582Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59827029Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598307311Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598402464Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598474176Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598507187Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/reload/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598514247Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/reload/reload/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598521227Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/reload/reload/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598570048Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/reload/reload/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598703271Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598888256Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/reload/reload/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598922587Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/telemeter-client/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598927637Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/telemeter-client/telemeter-client/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598931737Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.598974198Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599119762Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599184593Z namespaces/openshift-monitoring/pods/telemeter-client-f884997dd-7crk9/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599217254Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599290426Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/thanos-querier-67bb788c5-nt92b.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.59945201Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599471641Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599478891Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599490791Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599571833Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599649135Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599676176Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-rules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599682896Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599686856Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599766158Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599878791Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599946353Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599967943Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-web/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599976933Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.599985844Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600030335Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600134167Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600208269Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60024885Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600256791Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600267291Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600313822Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600403064Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600476356Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600496286Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/prom-label-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600507967Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600512687Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600560018Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60064912Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600743193Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600783754Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/thanos-query/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600799274Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/thanos-query/thanos-query/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600804134Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/thanos-query/thanos-query/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600843345Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.600943688Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601010979Z namespaces/openshift-monitoring/pods/thanos-querier-67bb788c5-nt92b/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60105542Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601095312Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601134353Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601174063Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601321027Z namespaces/openshift-multus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601365068Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60145279Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601493462Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601528663Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601570264Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60183836Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601917822Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.601995694Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602035305Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602075706Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60221846Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602341223Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602407355Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602448486Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602486767Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602528328Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60260593Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60264446Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602682232Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602790354Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602854776Z namespaces/openshift-multus/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.602892487Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60304241Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.603125633Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.603853361Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.603981624Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.604641781Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.604787234Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.604946369Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60501079Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.605068821Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.605206145Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.605313508Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60542162Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.605542653Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.605592045Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.605705448Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.605872522Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.605993335Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606059036Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606114988Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60619197Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606238581Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606314513Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606359164Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606434556Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606478957Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60659021Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606693992Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606796795Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606842986Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606922328Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.606967549Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.607044491Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.607090642Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.607162554Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.607172264Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.607242446Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6074041Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.607597885Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60778878Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608011226Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608217441Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608225271Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608287542Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/multus-additional-cni-plugins-57cvt.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608443256Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/bond-cni-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608488967Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608499168Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608567189Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608672332Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608866697Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608911598Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608918748Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/cni-plugins/cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.608925309Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60898757Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609248197Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609325579Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60936952Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/egress-router-binary-copy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60937698Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60938093Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609431541Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609550744Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609634556Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609673027Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609682637Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.60976059Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609830481Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.609918503Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610008776Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610049137Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/routeoverride-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610056357Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610060427Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610114908Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610205211Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610288393Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610343744Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni-bincopy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610350804Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610354725Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610423906Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610533549Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610634932Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610653192Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610662652Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610684273Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610774035Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.610891908Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61098711Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-57cvt/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611043902Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611115414Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/multus-additional-cni-plugins-9sl5f.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611256337Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/bond-cni-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611264697Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611268968Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611307949Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611397301Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611465252Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611495653Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611502153Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/cni-plugins/cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611524574Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611531564Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611622936Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611697218Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61175388Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/egress-router-binary-copy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61176668Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61177116Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611796571Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611890353Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611956895Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611975115Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.611993086Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612015456Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612027276Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612107369Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61217683Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612207491Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/routeoverride-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612215371Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612219261Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612258192Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612349325Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612415606Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612444097Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni-bincopy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612459917Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612466598Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612476938Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612582981Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612655072Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612667963Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612683913Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612698573Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612773025Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612853687Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612915959Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9sl5f/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61296606Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.612998521Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/multus-additional-cni-plugins-zxsfx.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613138104Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/bond-cni-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613145965Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613150215Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613195066Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613286568Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61335542Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613403951Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613411861Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/cni-plugins/cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613417151Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613464453Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613546335Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613614456Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613636847Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/egress-router-binary-copy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613644817Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613659377Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613692988Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613820041Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613892693Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613916064Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613922904Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613928864Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.613947175Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614036117Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614104539Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614122359Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/routeoverride-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614130839Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614134949Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614187141Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614298854Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614383406Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614426377Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni-bincopy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614437677Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614444317Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614496538Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614587371Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614668323Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614695674Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614705794Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614741545Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614819907Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614913189Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.614978801Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zxsfx/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.615017652Z namespaces/openshift-multus/pods/multus-j4jsd/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.615050472Z namespaces/openshift-multus/pods/multus-j4jsd/multus-j4jsd.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.615162255Z namespaces/openshift-multus/pods/multus-j4jsd/kube-multus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.615171875Z namespaces/openshift-multus/pods/multus-j4jsd/kube-multus/kube-multus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.615183396Z namespaces/openshift-multus/pods/multus-j4jsd/kube-multus/kube-multus/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.615224637Z namespaces/openshift-multus/pods/multus-j4jsd/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.616648853Z namespaces/openshift-multus/pods/multus-j4jsd/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.616761295Z namespaces/openshift-multus/pods/multus-j4jsd/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.616793986Z namespaces/openshift-multus/pods/multus-q4vcf/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.616857688Z namespaces/openshift-multus/pods/multus-q4vcf/multus-q4vcf.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61696234Z namespaces/openshift-multus/pods/multus-q4vcf/kube-multus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.616979861Z namespaces/openshift-multus/pods/multus-q4vcf/kube-multus/kube-multus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.616984411Z namespaces/openshift-multus/pods/multus-q4vcf/kube-multus/kube-multus/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.617011572Z namespaces/openshift-multus/pods/multus-q4vcf/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.617701809Z namespaces/openshift-multus/pods/multus-q4vcf/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.617848213Z namespaces/openshift-multus/pods/multus-q4vcf/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.617865983Z namespaces/openshift-multus/pods/multus-rrq8b/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.617945415Z namespaces/openshift-multus/pods/multus-rrq8b/multus-rrq8b.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.618050528Z namespaces/openshift-multus/pods/multus-rrq8b/kube-multus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.618063618Z namespaces/openshift-multus/pods/multus-rrq8b/kube-multus/kube-multus/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.618070368Z namespaces/openshift-multus/pods/multus-rrq8b/kube-multus/kube-multus/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.618115349Z namespaces/openshift-multus/pods/multus-rrq8b/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.618703775Z namespaces/openshift-multus/pods/multus-rrq8b/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.618812257Z namespaces/openshift-multus/pods/multus-rrq8b/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.618850468Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61891925Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/network-metrics-daemon-bx6mn.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619014332Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619027423Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619033923Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619078724Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619191097Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619275009Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61931391Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/network-metrics-daemon/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61932358Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.61934184Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619426082Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619684729Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619787082Z namespaces/openshift-multus/pods/network-metrics-daemon-bx6mn/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619814772Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619864124Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/network-metrics-daemon-fr9xb.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619951076Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619958716Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619962776Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.619995707Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620086899Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620156571Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620189742Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/network-metrics-daemon/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620197062Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620207112Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620264964Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620552201Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620646243Z namespaces/openshift-multus/pods/network-metrics-daemon-fr9xb/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620693854Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620748866Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/network-metrics-daemon-zvdqk.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620843818Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620851738Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620855698Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620887429Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.620986802Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621082134Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621119505Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/network-metrics-daemon/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621126285Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621132245Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621201757Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621349061Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621414443Z namespaces/openshift-multus/pods/network-metrics-daemon-zvdqk/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621442943Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621479944Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621547366Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621561066Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621569927Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621679909Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621759191Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621814363Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621917985Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.621979497Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.622036328Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6220944Z namespaces/openshift-must-gather-67kr8/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62210273Z namespaces/openshift-must-gather-67kr8/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62210899Z namespaces/openshift-must-gather-67kr8/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.622156251Z namespaces/openshift-must-gather-67kr8/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.622326855Z namespaces/openshift-must-gather-67kr8/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62251468Z namespaces/openshift-must-gather-67kr8/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.622662064Z namespaces/openshift-must-gather-67kr8/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.622983792Z namespaces/openshift-must-gather-67kr8/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623176217Z namespaces/openshift-network-console/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623226668Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62330953Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623346691Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623422803Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623462864Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623567957Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62369594Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623854514Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623899255Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.623950306Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624017598Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624059699Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62409633Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624134851Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624223603Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624272144Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624298805Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624394338Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624431848Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624473289Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624600303Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624690615Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624841729Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.624951372Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625116356Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625254729Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625389552Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625457854Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625501485Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625578837Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625622578Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62569754Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625793003Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625858694Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625893055Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625935966Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.625995548Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6260685Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626167362Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626215803Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626264394Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626331476Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626376457Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626445449Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626452949Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62649028Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626731666Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.626992733Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627147607Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627517406Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627731702Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627743972Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-fsxj8/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627798393Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-fsxj8/networking-console-plugin-74c9f884b7-fsxj8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627889315Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-fsxj8/networking-console-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627905266Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-fsxj8/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627912216Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-fsxj8/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.627923106Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-fsxj8/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628026739Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-fsxj8/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628114101Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-fsxj8/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628138062Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628216784Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628285205Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628310586Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628396308Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628413799Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628498481Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628538122Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628586753Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628643574Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628779958Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.628896001Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629020094Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629084425Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629144737Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629233319Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62927691Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629363092Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629413384Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629464925Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629502726Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629584448Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629614789Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.62965535Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629766162Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629823594Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629854035Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.629958238Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6300443Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.630249365Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.630326587Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.630531422Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.630616014Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.630731047Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.630878061Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.630905861Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.630972943Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631000084Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631096176Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631152197Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631170818Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631271351Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631312312Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631362003Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631394723Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631478666Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631558028Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631594819Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631621749Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631685601Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631733362Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631797124Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631839865Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631881546Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631888626Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.631931297Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632086581Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632266805Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632399509Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632613094Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632822959Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63283451Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-dbf99/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63286838Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-dbf99/network-check-source-7d88cb55d8-dbf99.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632938732Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-dbf99/check-endpoints/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632947573Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-dbf99/check-endpoints/check-endpoints/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632951973Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-dbf99/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.632992824Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-dbf99/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633110217Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-dbf99/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633178988Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-dbf99/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633201819Z namespaces/openshift-network-diagnostics/pods/network-check-target-67x96/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63324713Z namespaces/openshift-network-diagnostics/pods/network-check-target-67x96/network-check-target-67x96.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633465475Z namespaces/openshift-network-diagnostics/pods/network-check-target-67x96/network-check-target-container/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633473056Z namespaces/openshift-network-diagnostics/pods/network-check-target-67x96/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633476846Z namespaces/openshift-network-diagnostics/pods/network-check-target-67x96/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633515467Z namespaces/openshift-network-diagnostics/pods/network-check-target-67x96/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633598429Z namespaces/openshift-network-diagnostics/pods/network-check-target-67x96/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633674821Z namespaces/openshift-network-diagnostics/pods/network-check-target-67x96/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633690181Z namespaces/openshift-network-diagnostics/pods/network-check-target-qr9ct/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633780794Z namespaces/openshift-network-diagnostics/pods/network-check-target-qr9ct/network-check-target-qr9ct.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633867886Z namespaces/openshift-network-diagnostics/pods/network-check-target-qr9ct/network-check-target-container/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633877266Z namespaces/openshift-network-diagnostics/pods/network-check-target-qr9ct/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633881446Z namespaces/openshift-network-diagnostics/pods/network-check-target-qr9ct/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.633916897Z namespaces/openshift-network-diagnostics/pods/network-check-target-qr9ct/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634000829Z namespaces/openshift-network-diagnostics/pods/network-check-target-qr9ct/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634075811Z namespaces/openshift-network-diagnostics/pods/network-check-target-qr9ct/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634100342Z namespaces/openshift-network-diagnostics/pods/network-check-target-t542q/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634144603Z namespaces/openshift-network-diagnostics/pods/network-check-target-t542q/network-check-target-t542q.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634216245Z namespaces/openshift-network-diagnostics/pods/network-check-target-t542q/network-check-target-container/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634223054Z namespaces/openshift-network-diagnostics/pods/network-check-target-t542q/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634226985Z namespaces/openshift-network-diagnostics/pods/network-check-target-t542q/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634265976Z namespaces/openshift-network-diagnostics/pods/network-check-target-t542q/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634351858Z namespaces/openshift-network-diagnostics/pods/network-check-target-t542q/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63441573Z namespaces/openshift-network-diagnostics/pods/network-check-target-t542q/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63445864Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634496492Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634542563Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634550353Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634588804Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634672576Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634727427Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634785289Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634870671Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634921802Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.634964673Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635015434Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635057636Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635118077Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635158128Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635205009Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63524432Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635327833Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635408884Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635486317Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635519117Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635572259Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63561766Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635662341Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635763704Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635806514Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635853366Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635930228Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635975589Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.635986869Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63602249Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636068541Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636115362Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636209085Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636286106Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636371169Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636451631Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636531253Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636687097Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6368246Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636860891Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636908102Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.636958563Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637002945Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637049046Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637089367Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637126518Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637174129Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637253321Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637332493Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637376994Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637420225Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637465666Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637509307Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637552798Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637560199Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637595319Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637770264Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.637975649Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638115223Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638306747Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638492302Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638532533Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638582514Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638590804Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638627625Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638679447Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638723618Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638843621Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638883272Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638931163Z namespaces/openshift-network-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.638976544Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639040076Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639078007Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639125578Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639164439Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639269452Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639345454Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639427216Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639464746Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639510988Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639577339Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.63961473Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639661271Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639700382Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639805665Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639853256Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639893927Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.639975819Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64001334Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640023031Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640064732Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640109433Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640158984Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640278957Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640357699Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640519633Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640596635Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64079903Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.640882612Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641003725Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641058677Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641097467Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64118587Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641243251Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641289782Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641336093Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641378245Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641429246Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641474607Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64159507Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641635511Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641733294Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641840666Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641901918Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.641931248Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64199195Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642039881Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642086852Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642118433Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642175634Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642214455Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642262897Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642269727Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642310768Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642454681Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.642632866Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64279854Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643009755Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64319308Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64320007Z namespaces/openshift-network-operator/pods/iptables-alerter-45tdk/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643237001Z namespaces/openshift-network-operator/pods/iptables-alerter-45tdk/iptables-alerter-45tdk.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643314283Z namespaces/openshift-network-operator/pods/iptables-alerter-45tdk/iptables-alerter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643321703Z namespaces/openshift-network-operator/pods/iptables-alerter-45tdk/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643325854Z namespaces/openshift-network-operator/pods/iptables-alerter-45tdk/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643360655Z namespaces/openshift-network-operator/pods/iptables-alerter-45tdk/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643449137Z namespaces/openshift-network-operator/pods/iptables-alerter-45tdk/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643516999Z namespaces/openshift-network-operator/pods/iptables-alerter-45tdk/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643555079Z namespaces/openshift-network-operator/pods/iptables-alerter-bw7xm/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64358262Z namespaces/openshift-network-operator/pods/iptables-alerter-bw7xm/iptables-alerter-bw7xm.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643660602Z namespaces/openshift-network-operator/pods/iptables-alerter-bw7xm/iptables-alerter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643667912Z namespaces/openshift-network-operator/pods/iptables-alerter-bw7xm/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643671992Z namespaces/openshift-network-operator/pods/iptables-alerter-bw7xm/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643702693Z namespaces/openshift-network-operator/pods/iptables-alerter-bw7xm/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643818246Z namespaces/openshift-network-operator/pods/iptables-alerter-bw7xm/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643886758Z namespaces/openshift-network-operator/pods/iptables-alerter-bw7xm/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.643925539Z namespaces/openshift-network-operator/pods/iptables-alerter-k7jr9/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64396272Z namespaces/openshift-network-operator/pods/iptables-alerter-k7jr9/iptables-alerter-k7jr9.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644042872Z namespaces/openshift-network-operator/pods/iptables-alerter-k7jr9/iptables-alerter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644057082Z namespaces/openshift-network-operator/pods/iptables-alerter-k7jr9/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644062992Z namespaces/openshift-network-operator/pods/iptables-alerter-k7jr9/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644073942Z namespaces/openshift-network-operator/pods/iptables-alerter-k7jr9/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644160855Z namespaces/openshift-network-operator/pods/iptables-alerter-k7jr9/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644227706Z namespaces/openshift-network-operator/pods/iptables-alerter-k7jr9/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644266687Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644304428Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6443516Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644395781Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644439582Z namespaces/openshift-node/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644447322Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644451392Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644486693Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644635036Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644843972Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.644988525Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64518638Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645371545Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645415006Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645477488Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645519319Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6455683Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645601821Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645688023Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645805246Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645883878Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.645929759Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64596241Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646016571Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646043412Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646134664Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646163755Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646216376Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646296918Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646329389Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646381951Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646473943Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646591636Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646681468Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646852392Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.646932154Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647050527Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647348805Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647409856Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647454927Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647507019Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64754124Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647590491Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647637232Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647725104Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647821097Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647869268Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647906369Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64795464Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.647996171Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.648054302Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.648090753Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.648150675Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.648157715Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.648207636Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64836424Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.648543485Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.648678438Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.648905564Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649081728Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649116049Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649177211Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649210151Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649270703Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649294024Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649350275Z namespaces/openshift-operators/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649357435Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649361355Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649398276Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.64955328Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649757485Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.649910289Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650109244Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650296009Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.65033945Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650389621Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650434472Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650498864Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650538885Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650586146Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650616827Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650827692Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650903244Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.650976556Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651029837Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651079418Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651222462Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651295494Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651362045Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651413527Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651519389Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651578201Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651639203Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651897749Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.65194283Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.65194918Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.651997612Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.652055263Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.652111595Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.652350881Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.652447653Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.652857523Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.653001497Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654881614Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654913355Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654919585Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654924825Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654930025Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654934866Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654940346Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654946406Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654951696Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654956816Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654961756Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654966746Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654971587Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654976827Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654982027Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654986847Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.654991587Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655541671Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655563551Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655571191Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655577982Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655585172Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655593572Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655598672Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655609943Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655615633Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655643043Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655694675Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655790707Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655811017Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655826618Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.65591526Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.655978282Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.656016363Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.656085244Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.656093245Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.656122985Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.656281199Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.656466844Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.656601367Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657058749Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657316585Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657325386Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657395737Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovnkube-node-b92c2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657583232Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-node/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657592902Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657597522Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657620383Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657767207Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657856969Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6578955Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.65790315Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6579113Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.657943231Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658058984Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658124756Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658165127Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/nbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658177037Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/nbdb/nbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658182147Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/nbdb/nbdb/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658206988Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/nbdb/nbdb/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658315751Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658373762Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658392853Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/northd/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658399633Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/northd/northd/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658417123Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/northd/northd/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658442764Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/northd/northd/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658583547Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658663919Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/northd/northd/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658727691Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-acl-logging/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658744331Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658750521Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658795752Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658903805Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.658970447Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659013838Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659023668Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-controller/ovn-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659029649Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659039159Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.65950003Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659582312Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659619323Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovnkube-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659631204Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659689975Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.659774067Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662224269Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662304841Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662341602Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/sbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662351232Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/sbdb/sbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662357542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/sbdb/sbdb/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662404783Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/sbdb/sbdb/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662544347Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662635269Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b92c2/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.66267252Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.662869335Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovnkube-node-scdr5.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663024719Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-node/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663034559Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663040919Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663089621Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663204863Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663272395Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663313366Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663323406Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663329727Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663354157Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663494421Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663581993Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663604833Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/nbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663613444Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/nbdb/nbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663622124Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/nbdb/nbdb/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663660355Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/nbdb/nbdb/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663795368Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.66387605Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663912921Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/northd/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663920261Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/northd/northd/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663926162Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/northd/northd/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.663984223Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/northd/northd/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664087216Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664145607Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/northd/northd/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664197179Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-acl-logging/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664204969Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664211309Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664220479Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664301391Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664373123Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664413554Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664423724Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-controller/ovn-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664430054Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664439974Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664752562Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664837635Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664878516Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovnkube-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664886856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664896016Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.664943427Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.66665677Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.666759963Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.666810234Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/sbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.666820794Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/sbdb/sbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.666827505Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/sbdb/sbdb/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.666865266Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/sbdb/sbdb/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.666965218Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6670448Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-scdr5/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667084071Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667173403Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovnkube-node-v552f.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667370068Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-node/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667380639Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667385019Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667395079Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667510862Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667592974Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667642135Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667651185Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667657265Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667728227Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6678559Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667943853Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667984394Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/nbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667993404Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/nbdb/nbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.667999594Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/nbdb/nbdb/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668071126Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/nbdb/nbdb/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668189459Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668282541Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668336463Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/northd/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668347393Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/northd/northd/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668353493Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/northd/northd/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668400984Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/northd/northd/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668525827Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668607289Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/northd/northd/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.66864418Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-acl-logging/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668654331Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668670461Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668721562Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668842195Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668888726Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668898947Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-controller/ovn-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668903737Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.668937398Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.669324708Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.669403399Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.669451981Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovnkube-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.669459971Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.669466641Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.669507892Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.671634106Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.671728818Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.671768949Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/sbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.671778939Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/sbdb/sbdb/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67179497Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/sbdb/sbdb/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.671845421Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/sbdb/sbdb/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.671994644Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672007665Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v552f/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672048596Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672095247Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672124348Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672129888Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672171339Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672267641Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672394305Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672428035Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672479117Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672564919Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672667901Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672679682Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672762084Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672812365Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672820825Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672825765Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672882387Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672947399Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.672955289Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67301345Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.673225306Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.673449661Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.673611515Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.673867992Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674110928Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674155429Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674232061Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674292052Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674354074Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674406355Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674501498Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674654401Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674776485Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674819995Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674886927Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.674936429Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67499614Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675089032Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675146954Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675198385Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675293118Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675352499Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675359649Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67541233Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675472562Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675531093Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675653856Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67577213Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.675974704Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.676073717Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67619696Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.676331224Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.676511588Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67657972Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.676634681Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.676697933Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.676766655Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.676837026Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.676879197Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67698278Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677073612Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677135374Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677188885Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677251887Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677310408Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67737113Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677427691Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677487843Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677498613Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677546294Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677704558Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.677939454Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.678071757Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.678265172Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.678449257Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.678460377Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-6pcqx/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.678488938Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-6pcqx/service-ca-operator-5d6c75f84f-6pcqx.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67857899Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-6pcqx/service-ca-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67858619Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-6pcqx/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67859189Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-6pcqx/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.678648552Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-6pcqx/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.678867267Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-6pcqx/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679017971Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679049852Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679096593Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679138274Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679192825Z namespaces/openshift-service-ca/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679248677Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679316009Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.67936623Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679429811Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679478613Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679581095Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679856002Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.679965915Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680017876Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680077858Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680126859Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68018659Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680278553Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680337024Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680395826Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680486118Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680546099Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68055428Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680608171Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680667653Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680726524Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.680862127Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68095399Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.681116354Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.681194896Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68133765Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.681485423Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.681667218Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68174209Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.681793191Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.681867883Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.681908944Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.681974866Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682014507Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682128849Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682215012Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682274353Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682331744Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682388126Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682433387Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682495879Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682503449Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68255665Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682730874Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.682929819Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683077523Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683265938Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683448052Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683456583Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-kq5vf/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683496204Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-kq5vf/service-ca-658d7b7c78-kq5vf.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683591586Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-kq5vf/service-ca-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683599756Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-kq5vf/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683604136Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-kq5vf/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683655628Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-kq5vf/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.683950925Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-kq5vf/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684006926Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-kq5vf/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684076298Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68414412Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684184121Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684243123Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684273813Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684327305Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684381976Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684409827Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684464818Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684496239Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684580241Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684656193Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684775496Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684821427Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684884429Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68494299Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.684991921Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685094014Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685150375Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685204127Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685298289Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685357331Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685410692Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685532345Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685622307Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6857283Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685847923Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.685938785Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686075699Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686240003Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686346725Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686388526Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686460028Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686498409Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686569481Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686608182Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686741215Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686836168Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.686882459Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68694766Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687004202Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687047813Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687113035Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687124865Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687180206Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687351391Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687556356Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687696819Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.687979786Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688176222Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688216592Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688288434Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688329075Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688402947Z namespaces/openshift/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688447868Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68850197Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68853073Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688586262Z namespaces/openshift/apps/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688613733Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688728075Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688801947Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68890101Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.688952541Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689013092Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689063034Z namespaces/openshift/batch/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689123935Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689216748Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689275929Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.68932791Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689423453Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689480814Z namespaces/openshift/core/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689535576Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689650968Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689768662Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689880334Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.689968227Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.690067779Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.690207533Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.690391057Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.690445668Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.6905083Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.690559771Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.690623243Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.691963957Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.692011198Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.69212073Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.692217313Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.692336346Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.692454639Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.692559052Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.692746256Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.692915121Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.693052544Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.693162477Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.69326899Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.693401863Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.693524736Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.693647669Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.693815463Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.693928566Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.694050999Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.694175092Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.694304796Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.694451219Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.694572802Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.694685265Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.694919001Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695029334Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695122256Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695234679Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695328911Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695437234Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695533436Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695638059Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695780293Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695895485Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.695981868Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.69609101Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.696175943Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.696302256Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.69644765Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.696531032Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.696650824Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.696788438Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.696904821Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697034154Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697123876Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697215679Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697344232Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697475785Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697588618Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697699661Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697842465Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.697932317Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.698028809Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.698132582Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.698270055Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.698388958Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.698507421Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.698622514Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.698730347Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.69885183Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.698973053Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699050675Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699105916Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699213059Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699297131Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699332312Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699384983Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699437125Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699476395Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699526407Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699541137Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699585588Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699769733Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.699955918Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.700099101Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.700298696Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.700702036Z namespaces/openshift/policy/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.700777238Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.7008279Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.70085421Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.700907111Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.700917132Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.700950623Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.701105907Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.701218519Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.701323112Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.701433005Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.701591419Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.701705391Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.701848335Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.701959238Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.702082951Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.702191684Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.702314637Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.70244074Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.702578454Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.702685216Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.70283583Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.702946483Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703059376Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703171479Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703283161Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703389264Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703486976Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703591459Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703699722Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703836185Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.703943598Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704049901Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704162273Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704276226Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704385229Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704493362Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704594084Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704699297Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704870441Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.704979164Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.705080616Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.705183869Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.705281092Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.705397264Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.705501397Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.70561266Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.705763814Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.705919418Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.706041931Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.706168054Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.706295007Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.706443401Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.706575424Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.706923523Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707027476Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707145259Z network_logs/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707183399Z network_logs/cluster_scale [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707262681Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707343104Z network_logs/multi-networkpolicy [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707414575Z network_logs/net-attach-def [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707500517Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707642161Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.707765944Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.710522013Z nodes/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.710562874Z nodes/debug [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.710656837Z nodes/ip-10-0-133-215.ec2.internal/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.710704898Z nodes/ip-10-0-133-215.ec2.internal/cpu_affinities.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.711031406Z nodes/ip-10-0-133-215.ec2.internal/dmesg [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.711328754Z nodes/ip-10-0-133-215.ec2.internal/ethtool_channels [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.711399466Z nodes/ip-10-0-133-215.ec2.internal/ethtool_features [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.711490968Z nodes/ip-10-0-133-215.ec2.internal/ip-10-0-133-215.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.711915179Z nodes/ip-10-0-133-215.ec2.internal/irq_affinities.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.712073752Z nodes/ip-10-0-133-215.ec2.internal/lscpu [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.712197246Z nodes/ip-10-0-133-215.ec2.internal/lspci [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.712295908Z nodes/ip-10-0-133-215.ec2.internal/podresources.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.712400761Z nodes/ip-10-0-133-215.ec2.internal/pods_info.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.712516143Z nodes/ip-10-0-133-215.ec2.internal/proc_cmdline [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.712599056Z nodes/ip-10-0-133-215.ec2.internal/sysinfo.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.714317889Z nodes/ip-10-0-133-215.ec2.internal/sysinfo.tgz [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.714460283Z nodes/ip-10-0-136-111.ec2.internal/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.714493243Z nodes/ip-10-0-136-111.ec2.internal/cpu_affinities.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.714983696Z nodes/ip-10-0-136-111.ec2.internal/dmesg [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.715244022Z nodes/ip-10-0-136-111.ec2.internal/ethtool_channels [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.715318804Z nodes/ip-10-0-136-111.ec2.internal/ethtool_features [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.715399896Z nodes/ip-10-0-136-111.ec2.internal/ip-10-0-136-111.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.716018651Z nodes/ip-10-0-136-111.ec2.internal/irq_affinities.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.716165115Z nodes/ip-10-0-136-111.ec2.internal/lscpu [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.716284408Z nodes/ip-10-0-136-111.ec2.internal/lspci [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.7163697Z nodes/ip-10-0-136-111.ec2.internal/podresources.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.716480063Z nodes/ip-10-0-136-111.ec2.internal/pods_info.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.716601036Z nodes/ip-10-0-136-111.ec2.internal/proc_cmdline [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.716679978Z nodes/ip-10-0-136-111.ec2.internal/sysinfo.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.718414042Z nodes/ip-10-0-136-111.ec2.internal/sysinfo.tgz [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.718574346Z nodes/ip-10-0-137-238.ec2.internal/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.718612607Z nodes/ip-10-0-137-238.ec2.internal/cpu_affinities.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.71912066Z nodes/ip-10-0-137-238.ec2.internal/dmesg [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.719384006Z nodes/ip-10-0-137-238.ec2.internal/ethtool_channels [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.719457018Z nodes/ip-10-0-137-238.ec2.internal/ethtool_features [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.7195365Z nodes/ip-10-0-137-238.ec2.internal/ip-10-0-137-238.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.719994252Z nodes/ip-10-0-137-238.ec2.internal/irq_affinities.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.720143555Z nodes/ip-10-0-137-238.ec2.internal/lscpu [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.720267058Z nodes/ip-10-0-137-238.ec2.internal/lspci [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.720353891Z nodes/ip-10-0-137-238.ec2.internal/podresources.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.720470043Z nodes/ip-10-0-137-238.ec2.internal/pods_info.json [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.720579186Z nodes/ip-10-0-137-238.ec2.internal/proc_cmdline [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.720661728Z nodes/ip-10-0-137-238.ec2.internal/sysinfo.log [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.722434473Z nodes/ip-10-0-137-238.ec2.internal/sysinfo.tgz [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.722603137Z pod_network_connectivity_check/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.722632088Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.722777712Z static-pods/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.722790442Z static-pods/kube-apiserver/ [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.729121251Z [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.729147942Z sent 71,806 bytes received 9,580,074 bytes 6,434,586.67 bytes/sec [must-gather] [must-gather-k5rkf] OUT 2026-07-28T18:57:34.729156032Z total size is 128,114,076 speedup is 13.27 [must-gather] [must-gather ] OUT 2026-07-28T18:57:35.03289053Z namespace/openshift-must-gather-67kr8 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: ae3af18b-c19b-4ff5-9c6e-bc65d4c0a656 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-vfkkl [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-vfkkl-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-vfkkl [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] 7d63cb8..88ea44b ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Updating 7d63cb8..88ea44b [git-push-artifacts] Fast-forward [git-push-artifacts] .../e2e-llm-inference-service.tar.gz | Bin 0 -> 16722253 bytes [git-push-artifacts] .../maas-group-test-pb6pf/e2e-maas-openshift.tar.gz | Bin 0 -> 16651422 bytes [git-push-artifacts] 2 files changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-fspzw/e2e-llm-inference-service.tar.gz [git-push-artifacts] create mode 100644 test-artifacts/maas-group-test-pb6pf/e2e-maas-openshift.tar.gz [git-push-artifacts] -rw-r--r--. 1 root 1001540000 18612932 Jul 28 18:59 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-vfkkl/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts a4d9e45] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-vfkkl/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 88ea44b..a4d9e45 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-28T18:59:46.946Z","type":3}]